[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:12 dist-test-slave-2qp8 krb5kdc[4053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:12 dist-test-slave-2qp8 krb5kdc[4053](info): set up 2 sockets
Jun 19 04:45:12 dist-test-slave-2qp8 krb5kdc[4053](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 04:45:13.340633 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:13 dist-test-slave-2qp8 krb5kdc[4053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:13.512161 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:41409
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:13.901892 4066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:13.902175 4066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:13.902351 4066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:13.910504 4066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:13.910605 4066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:13.910689 4066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:13.910763 4066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:13.910826 4066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:13.920617 4066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:41409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:13.921945 4066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:13.923290 4066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:13.935587 4074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:13.935616 4072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:13.935822 4066 server_base.cc:1034] running on GCE node
W20260619 04:45:13.936067 4071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:13.937052 4066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:13.938380 4066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:13.939606 4066 hybrid_clock.cc:648] HybridClock initialized: now 1781844313939559 us; error 39 us; skew 500 ppm
I20260619 04:45:13.939957 4066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:13 dist-test-slave-2qp8 krb5kdc[4053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:13.943035 4066 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:13.944674 4066 webserver.cc:460] Webserver started at http://127.3.243.190:44009/ using document root <none> and password file <none>
I20260619 04:45:13.945348 4066 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:13.945492 4066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:13.945792 4066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:13.948768 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "1a4665a69ea44355a8f7fa53763f8b8c"
format_stamp: "Formatted at 2026-06-19 04:45:13 on dist-test-slave-2qp8"
server_key: "0969c02f9bf65166d222022bdc71267b"
server_key_iv: "c74e53962a3577c39c8e4b804152d657"
server_key_version: "encryptionkey@0"
I20260619 04:45:13.949501 4066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "1a4665a69ea44355a8f7fa53763f8b8c"
format_stamp: "Formatted at 2026-06-19 04:45:13 on dist-test-slave-2qp8"
server_key: "0969c02f9bf65166d222022bdc71267b"
server_key_iv: "c74e53962a3577c39c8e4b804152d657"
server_key_version: "encryptionkey@0"
I20260619 04:45:13.954726 4066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:45:13.958030 4081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:13.959306 4066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:13.959486 4066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "1a4665a69ea44355a8f7fa53763f8b8c"
format_stamp: "Formatted at 2026-06-19 04:45:13 on dist-test-slave-2qp8"
server_key: "0969c02f9bf65166d222022bdc71267b"
server_key_iv: "c74e53962a3577c39c8e4b804152d657"
server_key_version: "encryptionkey@0"
I20260619 04:45:13.959687 4066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:13.973922 4066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:13.977754 4066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:13.978040 4066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:13.996883 4066 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:41409
I20260619 04:45:13.996915 4132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:41409 every 8 connection(s)
I20260619 04:45:13.998289 4066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:14.001809 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 4066
I20260619 04:45:14.002149 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:14.002735 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:14.004741 4046 external_mini_cluster.cc:1428] Setting key 2343ea05b1dc7b4cf8082801f65b0c51
I20260619 04:45:14.015990 4133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c: Bootstrap starting.
I20260619 04:45:14.019235 4133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:14.020434 4133 log.cc:824] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:14.023933 4133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c: No bootstrap required, opened a new log
I20260619 04:45:14.033505 4133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4665a69ea44355a8f7fa53763f8b8c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41409 } }
I20260619 04:45:14.033850 4133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:14.034021 4133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4665a69ea44355a8f7fa53763f8b8c, State: Initialized, Role: FOLLOWER
I20260619 04:45:14.034744 4133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4665a69ea44355a8f7fa53763f8b8c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41409 } }
Jun 19 04:45:14 dist-test-slave-2qp8 krb5kdc[4053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:14.035039 4133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:14.035197 4133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:14.035415 4133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:14.038086 4133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4665a69ea44355a8f7fa53763f8b8c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41409 } }
I20260619 04:45:14.038619 4133 leader_election.cc:302] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a4665a69ea44355a8f7fa53763f8b8c; no voters:
I20260619 04:45:14.039305 4133 leader_election.cc:288] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:14.039479 4138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:14.040742 4138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [term 1 LEADER]: Becoming Leader. State: Replica: 1a4665a69ea44355a8f7fa53763f8b8c, State: Running, Role: LEADER
I20260619 04:45:14.041334 4138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4665a69ea44355a8f7fa53763f8b8c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41409 } }
I20260619 04:45:14.041738 4133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:14.044286 4139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a4665a69ea44355a8f7fa53763f8b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4665a69ea44355a8f7fa53763f8b8c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41409 } } }
I20260619 04:45:14.044701 4139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:14.049027 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:14.009440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58316 (local address 127.3.243.190:41409)
0619 04:45:14.011486 (+ 2046us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:14.011501 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:14.011556 (+ 55us) server_negotiation.cc:407] Connection header received
0619 04:45:14.020324 (+ 8768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:14.020346 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:14.020780 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:14.021261 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:14.025130 (+ 3869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:14.026461 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:14.028949 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:14.029649 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:14.029789 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:14.036756 (+ 6967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:14.036764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:14.036781 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:14.036822 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:14.041094 (+ 4272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:14.042200 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:14.042208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:14.042216 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:14.042434 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:14.043046 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:14.043051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:14.043054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:14.043999 (+ 945us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:14.044599 (+ 600us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:14.046423 (+ 1824us) server_negotiation.cc:299] Negotiation successful
0619 04:45:14.046979 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1814,"thread_start_us":1592,"threads_started":1}
I20260619 04:45:14.048477 4140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4665a69ea44355a8f7fa53763f8b8c. Latest consensus state: current_term: 1 leader_uuid: "1a4665a69ea44355a8f7fa53763f8b8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4665a69ea44355a8f7fa53763f8b8c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41409 } } }
I20260619 04:45:14.050544 4140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:14.051162 4143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:14.056454 4143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:14.066874 4143 catalog_manager.cc:1347] Generated new cluster ID: b7089331e2d54502bf77d62282c20f94
I20260619 04:45:14.067023 4143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:14.082856 4143 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:14.084259 4143 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:14.097694 4143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a4665a69ea44355a8f7fa53763f8b8c: Generated new TSK 0
I20260619 04:45:14.098557 4143 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:45:14.151733 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:41409
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:14.485884 4159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:14.486178 4159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:14.486338 4159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:14.493671 4159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:14.493818 4159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:14.494009 4159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:14.503852 4159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:41409
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:14.505232 4159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:14.506620 4159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:14.519685 4165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:14.519621 4167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:14.519862 4164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:14.520692 4159 server_base.cc:1034] running on GCE node
I20260619 04:45:14.521243 4159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:14.522199 4159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:14.523391 4159 hybrid_clock.cc:648] HybridClock initialized: now 1781844314523342 us; error 46 us; skew 500 ppm
I20260619 04:45:14.523717 4159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:14 dist-test-slave-2qp8 krb5kdc[4053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:14.526907 4159 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:45:14.528417 4159 webserver.cc:460] Webserver started at http://127.3.243.129:33467/ using document root <none> and password file <none>
I20260619 04:45:14.529071 4159 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:14.529213 4159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:14.529520 4159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:14.532334 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "e879addd6d544e1cb9b2593482e0807f"
format_stamp: "Formatted at 2026-06-19 04:45:14 on dist-test-slave-2qp8"
server_key: "e838904837e896f5477b18b10e364f89"
server_key_iv: "66a3d2459191132cbd6ea802e240543d"
server_key_version: "encryptionkey@0"
I20260619 04:45:14.533062 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "e879addd6d544e1cb9b2593482e0807f"
format_stamp: "Formatted at 2026-06-19 04:45:14 on dist-test-slave-2qp8"
server_key: "e838904837e896f5477b18b10e364f89"
server_key_iv: "66a3d2459191132cbd6ea802e240543d"
server_key_version: "encryptionkey@0"
I20260619 04:45:14.538043 4159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:45:14.541419 4174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:14.542759 4159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:45:14.542928 4159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "e879addd6d544e1cb9b2593482e0807f"
format_stamp: "Formatted at 2026-06-19 04:45:14 on dist-test-slave-2qp8"
server_key: "e838904837e896f5477b18b10e364f89"
server_key_iv: "66a3d2459191132cbd6ea802e240543d"
server_key_version: "encryptionkey@0"
I20260619 04:45:14.543190 4159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:14.582374 4159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:14.586822 4159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:14.587133 4159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:14.588145 4159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:14.589512 4159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:14.589617 4159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:14.589778 4159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:14.589851 4159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:14.624536 4159 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:46715
I20260619 04:45:14.624635 4286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:46715 every 8 connection(s)
I20260619 04:45:14.625882 4159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:14.630240 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 4159
I20260619 04:45:14.630482 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:14.631280 4046 external_mini_cluster.cc:1428] Setting key c212ba621dc2bcdf6d51329b241c65a3
Jun 19 04:45:14 dist-test-slave-2qp8 krb5kdc[4053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:45:14.649940 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:14.628595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:59221 (local address 127.3.243.190:41409)
0619 04:45:14.629068 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:14.629073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:14.629964 (+ 891us) server_negotiation.cc:407] Connection header received
0619 04:45:14.632489 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:14.632496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:14.632580 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:14.632683 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:14.634577 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:14.635314 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:14.637894 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:14.638523 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:14.638698 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:14.642304 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:14.642313 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:14.642321 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:14.642402 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:14.646442 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:14.647168 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:14.647174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:14.647178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:14.647343 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:14.647836 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:14.647843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:14.647847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:14.648299 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:14.648497 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:14.649202 (+ 705us) server_negotiation.cc:299] Negotiation successful
0619 04:45:14.649470 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":261,"threads_started":1}
I20260619 04:45:14.650586 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:14.629050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:41409 (local address 127.3.243.129:59221)
0619 04:45:14.629762 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:14.629814 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:14.632223 (+ 2409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:14.632950 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:14.632962 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:14.633618 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:14.634386 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:14.634411 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:14.635555 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:14.635562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:14.637731 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:14.637748 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:14.638743 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:14.638748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:14.638944 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:14.639641 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:14.639662 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:14.642033 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:14.646583 (+ 4550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:14.646589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:14.646601 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:14.647006 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:14.647512 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:14.647519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:14.647522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:14.647699 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:14.648443 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:14.648448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:14.648752 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 04:45:14.648993 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 04:45:14.649399 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":266,"threads_started":1}
I20260619 04:45:14.652207 4287 heartbeater.cc:347] Connected to a master server at 127.3.243.190:41409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
I20260619 04:45:14.652562 4287 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:14.653304 4287 heartbeater.cc:511] Master 127.3.243.190:41409 requested a full tablet report, sending...
I20260619 04:45:14.655527 4098 ts_manager.cc:194] Registered new tserver with Master: e879addd6d544e1cb9b2593482e0807f (127.3.243.129:46715)
I20260619 04:45:14.657632 4098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:59221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:45:14.674702 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:41409
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:14.995452 4293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:14.995788 4293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:14.995980 4293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:15.003425 4293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:15.003579 4293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:15.003772 4293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:15.013712 4293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:41409
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:15.015040 4293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:15.016471 4293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:15.029408 4298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:15.029488 4299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:15.029443 4301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:15.030546 4293 server_base.cc:1034] running on GCE node
I20260619 04:45:15.031172 4293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:15.032159 4293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:15.033407 4293 hybrid_clock.cc:648] HybridClock initialized: now 1781844315033344 us; error 56 us; skew 500 ppm
I20260619 04:45:15.033733 4293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:15.037073 4293 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:45:15.038645 4293 webserver.cc:460] Webserver started at http://127.3.243.130:39223/ using document root <none> and password file <none>
I20260619 04:45:15.039397 4293 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:15.039561 4293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:15.039902 4293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:15.043147 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "f6ca393d456a4df9bbcb391c30836d86"
format_stamp: "Formatted at 2026-06-19 04:45:15 on dist-test-slave-2qp8"
server_key: "85c743f15cda64d2d2dc5687f918e57b"
server_key_iv: "51510692cbd4723f656607f822787933"
server_key_version: "encryptionkey@0"
I20260619 04:45:15.043999 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "f6ca393d456a4df9bbcb391c30836d86"
format_stamp: "Formatted at 2026-06-19 04:45:15 on dist-test-slave-2qp8"
server_key: "85c743f15cda64d2d2dc5687f918e57b"
server_key_iv: "51510692cbd4723f656607f822787933"
server_key_version: "encryptionkey@0"
I20260619 04:45:15.049805 4293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:45:15.053331 4308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:15.054709 4293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 04:45:15.054873 4293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "f6ca393d456a4df9bbcb391c30836d86"
format_stamp: "Formatted at 2026-06-19 04:45:15 on dist-test-slave-2qp8"
server_key: "85c743f15cda64d2d2dc5687f918e57b"
server_key_iv: "51510692cbd4723f656607f822787933"
server_key_version: "encryptionkey@0"
I20260619 04:45:15.055079 4293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:15.078028 4293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:15.081964 4293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:15.082247 4293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:15.083189 4293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:15.084534 4293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:15.084609 4293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:15.084777 4293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:15.084828 4293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:15.117838 4293 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:40941
I20260619 04:45:15.117894 4420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:40941 every 8 connection(s)
I20260619 04:45:15.119180 4293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:15.120530 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 4293
I20260619 04:45:15.120757 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:15.121325 4046 external_mini_cluster.cc:1428] Setting key afed69db76f04ef8f8f67cadd332cf51
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:45:15.140635 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:15.121973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:35789 (local address 127.3.243.190:41409)
0619 04:45:15.122225 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:15.122231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:15.123519 (+ 1288us) server_negotiation.cc:407] Connection header received
0619 04:45:15.125553 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:15.125561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:15.125656 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:15.125763 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:15.127194 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.127943 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.129562 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.129995 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.130152 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:15.133940 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:15.133948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:15.133953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:15.134007 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:15.137526 (+ 3519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.138454 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.138460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.138463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.138640 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.139052 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.139055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.139057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.139313 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:15.139428 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:15.140229 (+ 801us) server_negotiation.cc:299] Negotiation successful
0619 04:45:15.140392 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
I20260619 04:45:15.142096 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:15.122542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:41409 (local address 127.3.243.130:35789)
0619 04:45:15.123351 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:15.123400 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:15.125285 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:15.125955 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:15.125963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:15.126470 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:15.127005 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:15.127032 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.128105 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.128110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:15.129401 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:15.129412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.130237 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.130244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:15.130479 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:15.131254 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:15.131288 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:15.133706 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:15.137857 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:15.137864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:15.137877 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:15.138310 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:15.138749 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:15.138753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:15.138755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:15.138938 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:15.139413 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:15.139419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:15.139839 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 04:45:15.140076 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 04:45:15.140622 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":333,"threads_started":1}
I20260619 04:45:15.143965 4421 heartbeater.cc:347] Connected to a master server at 127.3.243.190:41409
I20260619 04:45:15.144331 4421 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:15.145043 4421 heartbeater.cc:511] Master 127.3.243.190:41409 requested a full tablet report, sending...
I20260619 04:45:15.146668 4098 ts_manager.cc:194] Registered new tserver with Master: f6ca393d456a4df9bbcb391c30836d86 (127.3.243.130:40941)
I20260619 04:45:15.147979 4098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:35789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:45:15.163926 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:41409
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:15.506052 4426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:15.506356 4426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:15.506600 4426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:15.513916 4426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:15.514060 4426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:15.514241 4426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:45:15.524170 4426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:41409
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:15.525483 4426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:15.526885 4426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:15.540119 4432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:15.540409 4431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:15.540596 4434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:15.540743 4426 server_base.cc:1034] running on GCE node
I20260619 04:45:15.541431 4426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:15.542378 4426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:15.543608 4426 hybrid_clock.cc:648] HybridClock initialized: now 1781844315543554 us; error 49 us; skew 500 ppm
I20260619 04:45:15.543926 4426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:15.547170 4426 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:45:15.548651 4426 webserver.cc:460] Webserver started at http://127.3.243.131:36683/ using document root <none> and password file <none>
I20260619 04:45:15.549273 4426 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:15.549368 4426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:15.549623 4426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:15.552601 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "1bc396543478447c93221a46f3ee0760"
format_stamp: "Formatted at 2026-06-19 04:45:15 on dist-test-slave-2qp8"
server_key: "8fc6b03da31b048113877ac8c7cdc659"
server_key_iv: "d2ba4568dcb8c06857d652c8359c8b13"
server_key_version: "encryptionkey@0"
I20260619 04:45:15.553292 4426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "1bc396543478447c93221a46f3ee0760"
format_stamp: "Formatted at 2026-06-19 04:45:15 on dist-test-slave-2qp8"
server_key: "8fc6b03da31b048113877ac8c7cdc659"
server_key_iv: "d2ba4568dcb8c06857d652c8359c8b13"
server_key_version: "encryptionkey@0"
I20260619 04:45:15.558455 4426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:45:15.561911 4441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:15.563316 4426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:45:15.563475 4426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "1bc396543478447c93221a46f3ee0760"
format_stamp: "Formatted at 2026-06-19 04:45:15 on dist-test-slave-2qp8"
server_key: "8fc6b03da31b048113877ac8c7cdc659"
server_key_iv: "d2ba4568dcb8c06857d652c8359c8b13"
server_key_version: "encryptionkey@0"
I20260619 04:45:15.563668 4426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:15.582756 4426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:15.587066 4426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:15.587365 4426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:15.588361 4426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:15.589691 4426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:15.589763 4426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:15.589907 4426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:15.589956 4426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:15.622663 4426 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:42531
I20260619 04:45:15.622709 4553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:42531 every 8 connection(s)
I20260619 04:45:15.624035 4426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:45:15.631685 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 4426
I20260619 04:45:15.631908 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:45:15.632473 4046 external_mini_cluster.cc:1428] Setting key a5ec9a1789312eab39ad50e2ede7ec73
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:15.644603 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:15.626968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:45583 (local address 127.3.243.190:41409)
0619 04:45:15.627185 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:15.627191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:15.628367 (+ 1176us) server_negotiation.cc:407] Connection header received
0619 04:45:15.630332 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:15.630336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:15.630418 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:15.630511 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:15.632306 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.633107 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.634937 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.635397 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.635532 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:15.639175 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:15.639179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:15.639182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:15.639212 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:15.641878 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.642504 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.642507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.642510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.642633 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.643089 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.643093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.643112 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.643381 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:15.643498 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:15.644191 (+ 693us) server_negotiation.cc:299] Negotiation successful
0619 04:45:15.644349 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 04:45:15.645501 4556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:15.627357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:41409 (local address 127.3.243.131:45583)
0619 04:45:15.628192 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:15.628242 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:15.630072 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:15.630709 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:15.630716 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:15.631244 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:15.632133 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:15.632162 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.633288 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.633292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:15.634769 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:15.634780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.635632 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.635639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:15.635900 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:15.636816 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:15.636844 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:15.638985 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:15.641990 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:15.641996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:15.642008 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:15.642387 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:15.642769 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:15.642776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:15.642781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:15.642979 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:15.643483 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:15.643489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:15.643807 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 04:45:15.644034 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 04:45:15.644458 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":332,"threads_started":1}
I20260619 04:45:15.646997 4554 heartbeater.cc:347] Connected to a master server at 127.3.243.190:41409
I20260619 04:45:15.647372 4554 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:15.648288 4554 heartbeater.cc:511] Master 127.3.243.190:41409 requested a full tablet report, sending...
I20260619 04:45:15.649613 4098 ts_manager.cc:194] Registered new tserver with Master: 1bc396543478447c93221a46f3ee0760 (127.3.243.131:42531)
I20260619 04:45:15.650333 4098 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:45583
I20260619 04:45:15.660594 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:45:15.660578 4287 heartbeater.cc:503] Master 127.3.243.190:41409 was elected leader, sending a full tablet report...
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:45:15.705534 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:15.690012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.3.243.129:46715)
0619 04:45:15.690702 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:15.690712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:15.690745 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:45:15.690980 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:15.690987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:15.691205 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:15.691380 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:15.692922 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.693945 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.695215 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.695772 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.695908 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:15.699770 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:15.699780 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:15.699794 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:15.699823 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:15.702532 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.703178 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.703185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.703187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.703307 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.703717 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.703726 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.703728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.704053 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:15.704439 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:15.704733 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 04:45:15.705006 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":336,"threads_started":1}
W20260619 04:45:15.706876 4202 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49804
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:45:15.751811 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:15.737182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49814 (local address 127.3.243.129:46715)
0619 04:45:15.737438 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:15.737444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:15.737664 (+ 220us) server_negotiation.cc:407] Connection header received
0619 04:45:15.737811 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:15.737814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:15.737892 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:15.737979 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:15.739293 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.739956 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.741206 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.741638 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.741739 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:15.745799 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:15.745804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:15.745808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:15.745842 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:15.749009 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.749733 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.749741 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.749745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.749887 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.750306 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.750314 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.750318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.750607 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:15.750877 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:15.751207 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 04:45:15.751454 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 04:45:15.753484 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 4159
I20260619 04:45:15.766779 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:15.754594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.3.243.129:46715)
0619 04:45:15.754768 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:15.754772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:15.754980 (+ 208us) server_negotiation.cc:407] Connection header received
0619 04:45:15.755097 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:15.755101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:15.755158 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:15.755241 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:15.756455 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.757218 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.758510 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.759002 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.759105 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:15.761365 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:15.761369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:15.761372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:15.761401 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:15.764076 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.764828 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.764834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.764838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.764976 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.765343 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.765348 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.765351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.765621 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:15.765867 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:15.766183 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 04:45:15.766395 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:45:15.767556 4282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:15.854746 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 4159
I20260619 04:45:15.871232 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 4293
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:45:15.887014 4568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:15.872100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.3.243.130:40941)
0619 04:45:15.872748 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:15.872756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:15.872782 (+ 26us) server_negotiation.cc:407] Connection header received
0619 04:45:15.872844 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:15.872850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:15.873064 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:15.873194 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:15.874314 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.875326 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.876741 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.877276 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.877371 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:15.881265 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:15.881276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:15.881290 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:15.881330 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:15.884284 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.884953 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.884961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.884963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.885099 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:15.885485 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:15.885489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:15.885491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:15.885825 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:15.886054 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:15.886359 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 04:45:15.886628 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":318,"threads_started":1}
I20260619 04:45:15.888556 4416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:15.971438 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 4293
I20260619 04:45:15.986933 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 4426
Jun 19 04:45:15 dist-test-slave-2qp8 krb5kdc[4053](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:45:16.003768 4570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:15.987992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41902 (local address 127.3.243.131:42531)
0619 04:45:15.988677 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:15.988684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:15.988714 (+ 30us) server_negotiation.cc:407] Connection header received
0619 04:45:15.988776 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:15.988782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:15.989001 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:15.989140 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:15.990339 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.991341 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.992632 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:15.993239 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:15.993333 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:15.997449 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:15.997463 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:15.997483 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:15.997526 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:16.001000 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:16.001657 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:16.001667 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:16.001671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:16.001824 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:16.002164 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:16.002169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:16.002172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:16.002593 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:16.002877 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:16.003110 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 04:45:16.003368 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":365,"threads_started":1}
I20260619 04:45:16.005234 4549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:16.087443 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 4426
I20260619 04:45:16.103684 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 4066
I20260619 04:45:16.104992 4128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:16.210127 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 4066
2026-06-19T04:45:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3965 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:16 dist-test-slave-2qp8 krb5kdc[4575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:16 dist-test-slave-2qp8 krb5kdc[4575](info): set up 2 sockets
Jun 19 04:45:16 dist-test-slave-2qp8 krb5kdc[4575](info): commencing operation
krb5kdc: starting...
W20260619 04:45:17.296030 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:17 dist-test-slave-2qp8 krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:17.471459 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:37711
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:37711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:17.786017 4588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:17.786342 4588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:17.786530 4588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:17.793732 4588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:17.793874 4588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:17.793962 4588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:17.794070 4588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:17.794188 4588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:17.803983 4588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:37711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:37711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:17.805371 4588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:17.806712 4588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:17.819348 4594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:17.819382 4593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:17.819419 4596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:17.820618 4588 server_base.cc:1034] running on GCE node
I20260619 04:45:17.821441 4588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:17.822819 4588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:17.824059 4588 hybrid_clock.cc:648] HybridClock initialized: now 1781844317824018 us; error 77 us; skew 500 ppm
I20260619 04:45:17.824442 4588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:17 dist-test-slave-2qp8 krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844317, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:17.831312 4588 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:17.832841 4588 webserver.cc:460] Webserver started at http://127.3.243.190:35521/ using document root <none> and password file <none>
I20260619 04:45:17.833493 4588 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:17.833590 4588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:17.833833 4588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:17.836740 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "1431266ce67f42aaaa5fcf27883ac72e"
format_stamp: "Formatted at 2026-06-19 04:45:17 on dist-test-slave-2qp8"
server_key: "7771718e0019d0e562bf30b87efd58ed"
server_key_iv: "faceafadcda18ad1a679f0d894922e76"
server_key_version: "encryptionkey@0"
I20260619 04:45:17.837428 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "1431266ce67f42aaaa5fcf27883ac72e"
format_stamp: "Formatted at 2026-06-19 04:45:17 on dist-test-slave-2qp8"
server_key: "7771718e0019d0e562bf30b87efd58ed"
server_key_iv: "faceafadcda18ad1a679f0d894922e76"
server_key_version: "encryptionkey@0"
I20260619 04:45:17.842442 4588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 04:45:17.845880 4603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:17.847225 4588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:45:17.847371 4588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "1431266ce67f42aaaa5fcf27883ac72e"
format_stamp: "Formatted at 2026-06-19 04:45:17 on dist-test-slave-2qp8"
server_key: "7771718e0019d0e562bf30b87efd58ed"
server_key_iv: "faceafadcda18ad1a679f0d894922e76"
server_key_version: "encryptionkey@0"
I20260619 04:45:17.847548 4588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:17.859704 4588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:17.863605 4588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:17.863829 4588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:17.882237 4588 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:37711
I20260619 04:45:17.882259 4654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:37711 every 8 connection(s)
I20260619 04:45:17.883641 4588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:17.887398 4655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:17.888549 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 4588
I20260619 04:45:17.888773 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:17.889421 4046 external_mini_cluster.cc:1428] Setting key 5d5b5ba42a33facf48951a9254d772c7
I20260619 04:45:17.901528 4655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e: Bootstrap starting.
Jun 19 04:45:17 dist-test-slave-2qp8 krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:17.904738 4655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:17.906098 4655 log.cc:824] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:17.909602 4655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e: No bootstrap required, opened a new log
I20260619 04:45:17.912680 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:17.891376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.3.243.190:37711)
0619 04:45:17.892210 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:17.892230 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:17.892289 (+ 59us) server_negotiation.cc:407] Connection header received
0619 04:45:17.893962 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:17.893992 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:17.894519 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:17.894969 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:17.896372 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:17.898344 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:17.899604 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:17.900669 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:17.900842 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:17.904777 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:17.904789 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:17.904808 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:17.904858 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:17.908558 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:17.909195 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:17.909203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:17.909211 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:17.909406 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:17.909809 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:17.909813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:17.909815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:17.910332 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:17.910640 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:17.911147 (+ 507us) server_negotiation.cc:299] Negotiation successful
0619 04:45:17.911571 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":306,"threads_started":1}
I20260619 04:45:17.917521 4655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431266ce67f42aaaa5fcf27883ac72e" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37711 } }
I20260619 04:45:17.917886 4655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:17.917994 4655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1431266ce67f42aaaa5fcf27883ac72e, State: Initialized, Role: FOLLOWER
I20260619 04:45:17.918521 4655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431266ce67f42aaaa5fcf27883ac72e" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37711 } }
I20260619 04:45:17.918761 4655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:17.918913 4655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:17.919078 4655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:17.921396 4655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431266ce67f42aaaa5fcf27883ac72e" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37711 } }
I20260619 04:45:17.921905 4655 leader_election.cc:302] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1431266ce67f42aaaa5fcf27883ac72e; no voters:
I20260619 04:45:17.922447 4655 leader_election.cc:288] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:17.922606 4660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:17.923259 4660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [term 1 LEADER]: Becoming Leader. State: Replica: 1431266ce67f42aaaa5fcf27883ac72e, State: Running, Role: LEADER
I20260619 04:45:17.923735 4660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431266ce67f42aaaa5fcf27883ac72e" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37711 } }
I20260619 04:45:17.924516 4655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:17.926440 4662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1431266ce67f42aaaa5fcf27883ac72e. Latest consensus state: current_term: 1 leader_uuid: "1431266ce67f42aaaa5fcf27883ac72e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431266ce67f42aaaa5fcf27883ac72e" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37711 } } }
I20260619 04:45:17.926831 4662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:17.927524 4661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1431266ce67f42aaaa5fcf27883ac72e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1431266ce67f42aaaa5fcf27883ac72e" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37711 } } }
I20260619 04:45:17.927870 4661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e [sys.catalog]: This master's current role is: LEADER
W20260619 04:45:17.937356 4676 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:45:17.937512 4676 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:45:17.937646 4665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:17.939358 4665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:17.947791 4665 catalog_manager.cc:1347] Generated new cluster ID: 3f3eb36bb39a4d4f857e065cca3e94fa
I20260619 04:45:17.947913 4665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:17.969228 4665 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:17.970592 4665 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:17.979940 4665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1431266ce67f42aaaa5fcf27883ac72e: Generated new TSK 0
I20260619 04:45:17.981003 4665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:45:18.035625 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37711
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:18.378436 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:18.378721 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:18.378891 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:18.386121 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:18.386269 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:18.386467 4681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:18.396149 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:18.397478 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:18.398852 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:18.412389 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:18.412469 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:18.412663 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:18.413516 4681 server_base.cc:1034] running on GCE node
I20260619 04:45:18.414131 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:18.415117 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:18.416326 4681 hybrid_clock.cc:648] HybridClock initialized: now 1781844318416282 us; error 38 us; skew 500 ppm
I20260619 04:45:18.416651 4681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:18 dist-test-slave-2qp8 krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844318, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:18.419723 4681 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:45:18.421198 4681 webserver.cc:460] Webserver started at http://127.3.243.129:46029/ using document root <none> and password file <none>
I20260619 04:45:18.421880 4681 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:18.422049 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:18.422371 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:18.425520 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "8cf69984be8e40d9b918d462d6685123"
format_stamp: "Formatted at 2026-06-19 04:45:18 on dist-test-slave-2qp8"
server_key: "cde545a629672e5da15592bb604839f1"
server_key_iv: "9bd3981b74a158dd728e01b2081e8ce3"
server_key_version: "encryptionkey@0"
I20260619 04:45:18.426419 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "8cf69984be8e40d9b918d462d6685123"
format_stamp: "Formatted at 2026-06-19 04:45:18 on dist-test-slave-2qp8"
server_key: "cde545a629672e5da15592bb604839f1"
server_key_iv: "9bd3981b74a158dd728e01b2081e8ce3"
server_key_version: "encryptionkey@0"
I20260619 04:45:18.432215 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 04:45:18.435730 4696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:18.437263 4681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:18.437449 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "8cf69984be8e40d9b918d462d6685123"
format_stamp: "Formatted at 2026-06-19 04:45:18 on dist-test-slave-2qp8"
server_key: "cde545a629672e5da15592bb604839f1"
server_key_iv: "9bd3981b74a158dd728e01b2081e8ce3"
server_key_version: "encryptionkey@0"
I20260619 04:45:18.437666 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:18.458132 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:18.462330 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:18.462633 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:18.463634 4681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:18.465059 4681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:18.465168 4681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:18.465322 4681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:18.465405 4681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:18.498633 4681 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:33163
I20260619 04:45:18.498668 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:33163 every 8 connection(s)
I20260619 04:45:18.500043 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:18.504658 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 4681
I20260619 04:45:18.504899 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:18.505765 4046 external_mini_cluster.cc:1428] Setting key e7cf6f8c034d04778b7fb8914a6213db
Jun 19 04:45:18 dist-test-slave-2qp8 krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844318, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:45:18.524542 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:18.502952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:55917 (local address 127.3.243.190:37711)
0619 04:45:18.503391 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:18.503398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:18.504855 (+ 1457us) server_negotiation.cc:407] Connection header received
0619 04:45:18.507441 (+ 2586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:18.507446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:18.507541 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:18.507675 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:18.510070 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:18.510802 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:18.513591 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:18.514220 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:18.514365 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:18.518163 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:18.518169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:18.518172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:18.518208 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:18.521075 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:18.521945 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:18.521953 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:18.521961 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:18.522097 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:18.522572 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:18.522585 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:18.522587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:18.522862 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:18.523031 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:18.523852 (+ 821us) server_negotiation.cc:299] Negotiation successful
0619 04:45:18.524171 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":203,"threads_started":1}
I20260619 04:45:18.525698 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:18.503617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37711 (local address 127.3.243.129:55917)
0619 04:45:18.504602 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:18.504660 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:18.507146 (+ 2486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:18.508001 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:18.508024 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:18.508895 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:18.509835 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:18.509869 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:18.510978 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:18.510999 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:18.513370 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:18.513400 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:18.514475 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:18.514482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:18.514741 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:18.515464 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:18.515494 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:18.517858 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:18.521246 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:18.521256 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:18.521279 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:18.521790 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:18.522218 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:18.522223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:18.522227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:18.522439 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:18.522997 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:18.523006 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:18.523411 (+ 405us) client_negotiation.cc:769] Sending connection context
0619 04:45:18.523735 (+ 324us) client_negotiation.cc:240] Negotiation successful
0619 04:45:18.524373 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
I20260619 04:45:18.527447 4809 heartbeater.cc:347] Connected to a master server at 127.3.243.190:37711
I20260619 04:45:18.527797 4809 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:18.528637 4809 heartbeater.cc:511] Master 127.3.243.190:37711 requested a full tablet report, sending...
I20260619 04:45:18.531155 4620 ts_manager.cc:194] Registered new tserver with Master: 8cf69984be8e40d9b918d462d6685123 (127.3.243.129:33163)
I20260619 04:45:18.532876 4620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:55917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:45:18.548997 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37711
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:18.870616 4815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:18.870997 4815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:18.871193 4815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:18.878615 4815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:18.878770 4815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:18.878974 4815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:18.888816 4815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:18.890091 4815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:18.891485 4815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:18.904675 4821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:18.904803 4820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:18.904843 4823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:18.906198 4815 server_base.cc:1034] running on GCE node
I20260619 04:45:18.906775 4815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:18.907658 4815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:18.908854 4815 hybrid_clock.cc:648] HybridClock initialized: now 1781844318908801 us; error 42 us; skew 500 ppm
I20260619 04:45:18.909185 4815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:18 dist-test-slave-2qp8 krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844318, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:18.912290 4815 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:45:18.913803 4815 webserver.cc:460] Webserver started at http://127.3.243.130:32975/ using document root <none> and password file <none>
I20260619 04:45:18.914528 4815 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:18.914671 4815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:18.914992 4815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:18.917960 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "759f3a6508844d618661f5c925b1b41b"
format_stamp: "Formatted at 2026-06-19 04:45:18 on dist-test-slave-2qp8"
server_key: "4fdc52fb71c287fca1c2319d6066c601"
server_key_iv: "7ac7e6af6fb48ac5ed4d5853bbe79511"
server_key_version: "encryptionkey@0"
I20260619 04:45:18.918761 4815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "759f3a6508844d618661f5c925b1b41b"
format_stamp: "Formatted at 2026-06-19 04:45:18 on dist-test-slave-2qp8"
server_key: "4fdc52fb71c287fca1c2319d6066c601"
server_key_iv: "7ac7e6af6fb48ac5ed4d5853bbe79511"
server_key_version: "encryptionkey@0"
I20260619 04:45:18.923909 4815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 04:45:18.927615 4830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:18.929159 4815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:18.929327 4815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "759f3a6508844d618661f5c925b1b41b"
format_stamp: "Formatted at 2026-06-19 04:45:18 on dist-test-slave-2qp8"
server_key: "4fdc52fb71c287fca1c2319d6066c601"
server_key_iv: "7ac7e6af6fb48ac5ed4d5853bbe79511"
server_key_version: "encryptionkey@0"
I20260619 04:45:18.929540 4815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:18.944612 4815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:18.948851 4815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:18.949141 4815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:18.950125 4815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:18.951566 4815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:18.951637 4815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:18.951815 4815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:18.951889 4815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:18.982623 4815 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:37893
I20260619 04:45:18.982658 4942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:37893 every 8 connection(s)
I20260619 04:45:18.983922 4815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:18.986910 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 4815
I20260619 04:45:18.987119 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:18.987571 4046 external_mini_cluster.cc:1428] Setting key 65f678d15be8add68be81bb74a4cec2b
Jun 19 04:45:18 dist-test-slave-2qp8 krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844318, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:45:19.005180 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:18.986547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:55553 (local address 127.3.243.190:37711)
0619 04:45:18.986782 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:18.986790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:18.988153 (+ 1363us) server_negotiation.cc:407] Connection header received
0619 04:45:18.990038 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:18.990044 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:18.990128 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:18.990248 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:18.991666 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:18.992817 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:18.994755 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:18.995163 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:18.995261 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:18.999393 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:18.999398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:18.999401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:18.999432 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:19.002126 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.002848 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.002855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.002858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.002980 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.003404 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.003411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.003415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.003682 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:19.003850 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.004598 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.004830 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 04:45:19.006129 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:18.987113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37711 (local address 127.3.243.130:55553)
0619 04:45:18.987966 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:18.988015 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:18.989778 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:18.990434 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:18.990442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:18.990956 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:18.991487 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:18.991503 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:18.992955 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:18.992959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:18.994606 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:18.994621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:18.995379 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:18.995388 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:18.995782 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:18.996524 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:18.996556 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:18.999167 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:19.002265 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:19.002274 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:19.002292 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:19.002722 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:19.003091 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:19.003095 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:19.003099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:19.003295 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:19.003811 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:19.003820 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:19.004250 (+ 430us) client_negotiation.cc:769] Sending connection context
0619 04:45:19.004519 (+ 269us) client_negotiation.cc:240] Negotiation successful
0619 04:45:19.005058 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":350,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
I20260619 04:45:19.007704 4943 heartbeater.cc:347] Connected to a master server at 127.3.243.190:37711
I20260619 04:45:19.008040 4943 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:19.008790 4943 heartbeater.cc:511] Master 127.3.243.190:37711 requested a full tablet report, sending...
I20260619 04:45:19.010344 4620 ts_manager.cc:194] Registered new tserver with Master: 759f3a6508844d618661f5c925b1b41b (127.3.243.130:37893)
I20260619 04:45:19.011211 4620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:55553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:45:19.029171 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37711
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:19.352288 4948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:19.352612 4948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:19.352780 4948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:19.360076 4948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:19.360285 4948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:19.360484 4948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:45:19.370401 4948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37711
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:19.371717 4948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:19.373132 4948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:19.385967 4953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:19.386025 4956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:19.386049 4954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:19.386955 4948 server_base.cc:1034] running on GCE node
I20260619 04:45:19.387543 4948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:19.388562 4948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:19.389756 4948 hybrid_clock.cc:648] HybridClock initialized: now 1781844319389723 us; error 32 us; skew 500 ppm
I20260619 04:45:19.390080 4948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:19 dist-test-slave-2qp8 krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844319, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:19.393155 4948 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:45:19.394608 4948 webserver.cc:460] Webserver started at http://127.3.243.131:39461/ using document root <none> and password file <none>
I20260619 04:45:19.395315 4948 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:19.395443 4948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:19.395757 4948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:19.398648 4948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "7bc96d2ed8c4431690c976c1bbec4b78"
format_stamp: "Formatted at 2026-06-19 04:45:19 on dist-test-slave-2qp8"
server_key: "b0d8595ed788372dae594932dfa5c150"
server_key_iv: "54bd59b0816b8513166afccff185e63d"
server_key_version: "encryptionkey@0"
I20260619 04:45:19.399449 4948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "7bc96d2ed8c4431690c976c1bbec4b78"
format_stamp: "Formatted at 2026-06-19 04:45:19 on dist-test-slave-2qp8"
server_key: "b0d8595ed788372dae594932dfa5c150"
server_key_iv: "54bd59b0816b8513166afccff185e63d"
server_key_version: "encryptionkey@0"
I20260619 04:45:19.404621 4948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 04:45:19.407914 4963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:19.409291 4948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:45:19.409467 4948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "7bc96d2ed8c4431690c976c1bbec4b78"
format_stamp: "Formatted at 2026-06-19 04:45:19 on dist-test-slave-2qp8"
server_key: "b0d8595ed788372dae594932dfa5c150"
server_key_iv: "54bd59b0816b8513166afccff185e63d"
server_key_version: "encryptionkey@0"
I20260619 04:45:19.409680 4948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:19.428246 4948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:19.432512 4948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:19.432782 4948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:19.433727 4948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:19.435060 4948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:19.435160 4948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:19.435308 4948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:19.435379 4948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:19.468048 4948 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:37755
I20260619 04:45:19.468094 5075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:37755 every 8 connection(s)
I20260619 04:45:19.469372 4948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:45:19.476553 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 4948
I20260619 04:45:19.476758 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:45:19.477295 4046 external_mini_cluster.cc:1428] Setting key 9af27374fda21d0784736318f58feb7a
Jun 19 04:45:19 dist-test-slave-2qp8 krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844319, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:19.490093 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.471827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:39895 (local address 127.3.243.190:37711)
0619 04:45:19.472048 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.472053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.473211 (+ 1158us) server_negotiation.cc:407] Connection header received
0619 04:45:19.474993 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.474998 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.475070 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.475159 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:19.476727 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.477821 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.479937 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.480558 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.480666 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.483974 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:19.483978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:19.483981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:19.484021 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:19.487073 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.487854 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.487859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.487863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.487994 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.488432 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.488439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.488442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.488708 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:19.488861 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.489526 (+ 665us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.489727 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 04:45:19.491122 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.472375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37711 (local address 127.3.243.131:39895)
0619 04:45:19.473047 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:19.473083 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:19.474724 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:19.475363 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:19.475371 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:19.475871 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:19.476531 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.476550 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.477982 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.477986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.479802 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.479816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.480831 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.480837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.481053 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.481676 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:19.481702 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:19.483793 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:19.487198 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:19.487206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:19.487222 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:19.487727 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:19.488125 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:19.488132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:19.488135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:19.488320 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:19.488835 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:19.488842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:19.489169 (+ 327us) client_negotiation.cc:769] Sending connection context
0619 04:45:19.489435 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 04:45:19.489946 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":253,"threads_started":1}
I20260619 04:45:19.492699 5076 heartbeater.cc:347] Connected to a master server at 127.3.243.190:37711
I20260619 04:45:19.492942 5076 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:19.493577 5076 heartbeater.cc:511] Master 127.3.243.190:37711 requested a full tablet report, sending...
I20260619 04:45:19.494837 4620 ts_manager.cc:194] Registered new tserver with Master: 7bc96d2ed8c4431690c976c1bbec4b78 (127.3.243.131:37755)
I20260619 04:45:19.495572 4620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:39895
I20260619 04:45:19.504827 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:45:19.523010 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.511170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.3.243.190:37711)
0619 04:45:19.511419 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.511427 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.511767 (+ 340us) server_negotiation.cc:407] Connection header received
0619 04:45:19.511830 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.511834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.511901 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.511989 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:19.513482 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.514363 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.515681 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.516064 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.516229 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.518378 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:19.518382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:19.518385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:19.518411 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:19.520861 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.521400 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.521403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.521405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.521534 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.521820 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.521823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.521824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.522096 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:19.522261 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.522523 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.522672 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 04:45:19.529296 4620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:45:19.536573 4620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:19.537031 4809 heartbeater.cc:503] Master 127.3.243.190:37711 was elected leader, sending a full tablet report...
I20260619 04:45:19.559280 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.549037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39024 (local address 127.3.243.131:37755)
0619 04:45:19.549684 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.549696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.549737 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:45:19.549854 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.549865 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.550163 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.551233 (+ 1070us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:19.551639 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.553762 (+ 2123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.556645 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.558033 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.558233 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.558469 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.558752 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.558920 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":409,"threads_started":1}
I20260619 04:45:19.559301 5088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.548336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:37755 (local address 127.0.0.1:39024)
0619 04:45:19.549279 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:19.549330 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:19.549480 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:19.550493 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:19.550516 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:19.550557 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:19.550998 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.551011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.553940 (+ 2929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.553949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.556374 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.556390 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.558296 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.558302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.558491 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.558555 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 04:45:19.558685 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 04:45:19.558873 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":38,"thread_start_us":588,"threads_started":1}
I20260619 04:45:19.561084 5091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.548547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:37893 (local address 127.0.0.1:35342)
0619 04:45:19.550188 (+ 1641us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:19.550209 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:19.550319 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:19.553199 (+ 2880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:19.553204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:19.553225 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:19.553625 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.553638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.555347 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.555373 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.557666 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.557679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.560286 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.560291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.560396 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.560453 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 04:45:19.560530 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:45:19.560665 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1500,"mutex_wait_us":3,"thread_start_us":4102,"threads_started":1}
I20260619 04:45:19.561311 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.550561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57312 (local address 127.3.243.129:33163)
0619 04:45:19.551194 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.551205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.552971 (+ 1766us) server_negotiation.cc:407] Connection header received
0619 04:45:19.553243 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.553252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.553474 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.553648 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:19.554636 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.556146 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.558149 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.559996 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.560141 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.560353 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.560811 (+ 458us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.560962 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":338,"threads_started":1}
I20260619 04:45:19.561540 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.549699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33163 (local address 127.0.0.1:57312)
0619 04:45:19.552868 (+ 3169us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:19.552885 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:19.553064 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:19.553915 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:19.553919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:19.553940 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:19.554416 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.554432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.556308 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.556316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.557983 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.557996 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.560498 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.560505 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.560642 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.560663 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 04:45:19.560774 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 04:45:19.560895 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3064,"thread_start_us":1375,"threads_started":1}
I20260619 04:45:19.562170 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.550030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.3.243.130:37893)
0619 04:45:19.552583 (+ 2553us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.552594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.552625 (+ 31us) server_negotiation.cc:407] Connection header received
0619 04:45:19.552730 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.552738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.552989 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.553804 (+ 815us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:19.554027 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.555223 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.557847 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.560033 (+ 2186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.561540 (+ 1507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.561785 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.561848 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.561966 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2396,"thread_start_us":2275,"threads_started":1}
I20260619 04:45:19.563544 5011 tablet_service.cc:1467] Processing CreateTablet for tablet c983a561e0764a46918ad5ad92d3d48b (DEFAULT_TABLE table=test-table [id=a58690296b354abfb1c58fda1c0b7ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:19.565147 5011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c983a561e0764a46918ad5ad92d3d48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:19.565056 4744 tablet_service.cc:1467] Processing CreateTablet for tablet c983a561e0764a46918ad5ad92d3d48b (DEFAULT_TABLE table=test-table [id=a58690296b354abfb1c58fda1c0b7ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:19.565541 4878 tablet_service.cc:1467] Processing CreateTablet for tablet c983a561e0764a46918ad5ad92d3d48b (DEFAULT_TABLE table=test-table [id=a58690296b354abfb1c58fda1c0b7ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:19.566334 4744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c983a561e0764a46918ad5ad92d3d48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:19.566825 4878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c983a561e0764a46918ad5ad92d3d48b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:19.577303 5094 tablet_bootstrap.cc:492] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123: Bootstrap starting.
I20260619 04:45:19.579526 5095 tablet_bootstrap.cc:492] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78: Bootstrap starting.
I20260619 04:45:19.581182 5094 tablet_bootstrap.cc:654] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:19.582614 5094 log.cc:824] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:19.583606 5095 tablet_bootstrap.cc:654] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:19.584836 5095 log.cc:824] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:19.585855 5094 tablet_bootstrap.cc:492] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123: No bootstrap required, opened a new log
I20260619 04:45:19.586112 5094 ts_tablet_manager.cc:1397] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:45:19.587663 5095 tablet_bootstrap.cc:492] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78: No bootstrap required, opened a new log
I20260619 04:45:19.587817 5096 tablet_bootstrap.cc:492] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b: Bootstrap starting.
I20260619 04:45:19.587918 5095 ts_tablet_manager.cc:1397] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:45:19.590662 5096 tablet_bootstrap.cc:654] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:19.591686 5096 log.cc:824] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:19.594048 5096 tablet_bootstrap.cc:492] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b: No bootstrap required, opened a new log
I20260619 04:45:19.594290 5096 ts_tablet_manager.cc:1397] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 04:45:19.596691 5094 raft_consensus.cc:348] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } }
I20260619 04:45:19.597138 5094 raft_consensus.cc:374] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:19.597239 5094 raft_consensus.cc:729] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf69984be8e40d9b918d462d6685123, State: Initialized, Role: FOLLOWER
I20260619 04:45:19.597867 5094 consensus_queue.cc:260] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } }
I20260619 04:45:19.598368 5095 raft_consensus.cc:348] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } }
I20260619 04:45:19.598783 5095 raft_consensus.cc:374] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:19.598875 5095 raft_consensus.cc:729] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc96d2ed8c4431690c976c1bbec4b78, State: Initialized, Role: FOLLOWER
I20260619 04:45:19.599642 5094 ts_tablet_manager.cc:1428] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 04:45:19.599543 5095 consensus_queue.cc:260] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } }
I20260619 04:45:19.600950 5076 heartbeater.cc:503] Master 127.3.243.190:37711 was elected leader, sending a full tablet report...
I20260619 04:45:19.601542 5095 ts_tablet_manager.cc:1428] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260619 04:45:19.602797 5096 raft_consensus.cc:348] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } }
I20260619 04:45:19.603207 5096 raft_consensus.cc:374] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:19.603310 5096 raft_consensus.cc:729] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759f3a6508844d618661f5c925b1b41b, State: Initialized, Role: FOLLOWER
I20260619 04:45:19.603972 5096 consensus_queue.cc:260] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } }
I20260619 04:45:19.606061 4943 heartbeater.cc:503] Master 127.3.243.190:37711 was elected leader, sending a full tablet report...
I20260619 04:45:19.606590 5096 ts_tablet_manager.cc:1428] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 04:45:19.621819 5101 raft_consensus.cc:482] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:19.622043 5101 raft_consensus.cc:504] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } }
I20260619 04:45:19.623606 5101 leader_election.cc:288] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 759f3a6508844d618661f5c925b1b41b (127.3.243.130:37893), 8cf69984be8e40d9b918d462d6685123 (127.3.243.129:33163)
I20260619 04:45:19.629133 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.623975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:42903 (local address 127.3.243.129:33163)
0619 04:45:19.624204 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.624210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.624277 (+ 67us) server_negotiation.cc:407] Connection header received
0619 04:45:19.624527 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.624533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.624603 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.624695 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:19.625136 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.625940 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.627455 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.628294 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.628452 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.628536 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.628771 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.628890 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:45:19.629200 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.623943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33163 (local address 127.3.243.131:42903)
0619 04:45:19.624186 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:19.624205 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:19.624340 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:19.624715 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:19.624720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:19.624751 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:19.624962 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.624970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.626082 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.626089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.627311 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.627322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.628439 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.628445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.628621 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.628645 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 04:45:19.628748 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 04:45:19.628873 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:45:19.630026 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.624011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:55251 (local address 127.3.243.130:37893)
0619 04:45:19.624219 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.624226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.625086 (+ 860us) server_negotiation.cc:407] Connection header received
0619 04:45:19.625273 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.625283 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.625337 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.625425 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:19.625943 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.626601 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.628184 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.628927 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.629116 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.629181 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.629612 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.629737 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:45:19.630089 5103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.623966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:37893 (local address 127.3.243.131:55251)
0619 04:45:19.624984 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:19.624998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:19.625137 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:19.625435 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:19.625440 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:19.625460 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:19.625772 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.625781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.626737 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.626742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.628016 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:19.628027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.629139 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.629144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:19.629455 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.629480 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 04:45:19.629589 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 04:45:19.629705 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":244,"thread_start_us":438,"threads_started":1}
I20260619 04:45:19.630198 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c983a561e0764a46918ad5ad92d3d48b" candidate_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf69984be8e40d9b918d462d6685123" is_pre_election: true
I20260619 04:45:19.630780 4764 raft_consensus.cc:2454] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bc96d2ed8c4431690c976c1bbec4b78 in term 0.
I20260619 04:45:19.631035 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c983a561e0764a46918ad5ad92d3d48b" candidate_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759f3a6508844d618661f5c925b1b41b" is_pre_election: true
I20260619 04:45:19.631551 4966 leader_election.cc:302] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bc96d2ed8c4431690c976c1bbec4b78, 8cf69984be8e40d9b918d462d6685123; no voters:
I20260619 04:45:19.631620 4898 raft_consensus.cc:2454] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7bc96d2ed8c4431690c976c1bbec4b78 in term 0.
I20260619 04:45:19.631929 5101 raft_consensus.cc:2781] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:19.632085 5101 raft_consensus.cc:482] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:19.632229 5101 raft_consensus.cc:3037] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:19.634451 5101 raft_consensus.cc:504] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } }
I20260619 04:45:19.635007 5101 leader_election.cc:288] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [CANDIDATE]: Term 1 election: Requested vote from peers 759f3a6508844d618661f5c925b1b41b (127.3.243.130:37893), 8cf69984be8e40d9b918d462d6685123 (127.3.243.129:33163)
I20260619 04:45:19.635361 4898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c983a561e0764a46918ad5ad92d3d48b" candidate_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "759f3a6508844d618661f5c925b1b41b"
I20260619 04:45:19.635484 4764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c983a561e0764a46918ad5ad92d3d48b" candidate_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cf69984be8e40d9b918d462d6685123"
I20260619 04:45:19.635567 4898 raft_consensus.cc:3037] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:19.635684 4764 raft_consensus.cc:3037] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:19.638865 4898 raft_consensus.cc:2454] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bc96d2ed8c4431690c976c1bbec4b78 in term 1.
I20260619 04:45:19.638865 4764 raft_consensus.cc:2454] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7bc96d2ed8c4431690c976c1bbec4b78 in term 1.
I20260619 04:45:19.639251 4966 leader_election.cc:302] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bc96d2ed8c4431690c976c1bbec4b78, 8cf69984be8e40d9b918d462d6685123; no voters:
I20260619 04:45:19.639557 5101 raft_consensus.cc:2781] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:19.640074 5101 raft_consensus.cc:686] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [term 1 LEADER]: Becoming Leader. State: Replica: 7bc96d2ed8c4431690c976c1bbec4b78, State: Running, Role: LEADER
I20260619 04:45:19.640559 5101 consensus_queue.cc:237] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } }
I20260619 04:45:19.644953 4618 catalog_manager.cc:5495] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7bc96d2ed8c4431690c976c1bbec4b78 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bc96d2ed8c4431690c976c1bbec4b78" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "759f3a6508844d618661f5c925b1b41b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cf69984be8e40d9b918d462d6685123" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33163 } health_report { overall_health: UNKNOWN } } }
Jun 19 04:45:19 dist-test-slave-2qp8 krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:45:19.668452 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.653895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57316 (local address 127.3.243.129:33163)
0619 04:45:19.654231 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.654239 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.654773 (+ 534us) server_negotiation.cc:407] Connection header received
0619 04:45:19.654935 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.654939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.655006 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.655103 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:19.656692 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.657448 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.658680 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.659089 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.659176 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.663147 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:19.663160 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:19.663177 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:19.663210 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:19.666115 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.666696 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.666702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.666704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.666819 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.667131 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.667133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.667135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.667459 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:19.667681 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.667879 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.668032 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
Jun 19 04:45:19 dist-test-slave-2qp8 krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844319, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:45:19 dist-test-slave-2qp8 krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844319, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:45:19.712275 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.699499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.3.243.129:33163)
0619 04:45:19.699711 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.699718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.700035 (+ 317us) server_negotiation.cc:407] Connection header received
0619 04:45:19.700260 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.700264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.700346 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.700437 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:19.701704 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.702420 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.703597 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.703977 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.704061 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.707809 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:19.707814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:19.707818 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:19.707850 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:19.710217 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.710747 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.710751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.710753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.710872 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.711195 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.711198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.711200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.711446 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:19.711629 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.711819 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.711965 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 04:45:19.721027 5077 tablet.cc:2369] T c983a561e0764a46918ad5ad92d3d48b P 7bc96d2ed8c4431690c976c1bbec4b78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:45:19.735824 4944 tablet.cc:2369] T c983a561e0764a46918ad5ad92d3d48b P 759f3a6508844d618661f5c925b1b41b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 04:45:19 dist-test-slave-2qp8 krb5kdc[4575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:45:19 dist-test-slave-2qp8 krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
W20260619 04:45:19.752391 4810 tablet.cc:2369] T c983a561e0764a46918ad5ad92d3d48b P 8cf69984be8e40d9b918d462d6685123: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:45:19.756767 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.743596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57336 (local address 127.3.243.129:33163)
0619 04:45:19.743824 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.743828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.743936 (+ 108us) server_negotiation.cc:407] Connection header received
0619 04:45:19.744161 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.744166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.744233 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.744322 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:19.745632 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.746297 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.747435 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.747882 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.747973 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.751713 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:19.751718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:19.751721 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:19.751753 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:19.754621 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.755198 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.755202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.755204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.755323 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.755606 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.755610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.755611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.755870 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:19.756060 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.756344 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.756521 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260619 04:45:19.758433 4724 tablet_service.cc:2908] Rejecting scan request for tablet c983a561e0764a46918ad5ad92d3d48b: Uninitialized: clean time has not yet been initialized
I20260619 04:45:19.761337 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 4681
I20260619 04:45:19.772642 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.762196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57352 (local address 127.3.243.129:33163)
0619 04:45:19.762389 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.762397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.762670 (+ 273us) server_negotiation.cc:407] Connection header received
0619 04:45:19.762768 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.762772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.762835 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.762909 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:19.763951 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.764678 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.765807 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.766173 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.766267 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.768292 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:19.768297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:19.768300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:19.768326 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:19.770531 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.771086 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.771090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.771092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.771195 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.771472 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.771475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.771476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.771738 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:19.771934 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.772163 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.772314 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:45:19.773248 4804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:19.863801 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 4681
I20260619 04:45:19.880323 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 4815
Jun 19 04:45:19 dist-test-slave-2qp8 krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:45:19.896715 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:19.881153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35356 (local address 127.3.243.130:37893)
0619 04:45:19.881356 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:19.881360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:19.881378 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:45:19.881544 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:19.881549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:19.881641 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:19.881724 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:19.883189 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.883988 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.885808 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:19.886204 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:19.886297 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:19.891036 (+ 4739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:19.891051 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:19.891067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:19.891101 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:19.894074 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.894808 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.894815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.894817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.894945 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:19.895315 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:19.895318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:19.895320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:19.895672 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:19.895909 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:19.896255 (+ 346us) server_negotiation.cc:299] Negotiation successful
0619 04:45:19.896410 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 04:45:19.897475 4938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:19.983596 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 4815
I20260619 04:45:19.999392 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 4948
Jun 19 04:45:20 dist-test-slave-2qp8 krb5kdc[4575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:45:20.014003 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:20.000259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39032 (local address 127.3.243.131:37755)
0619 04:45:20.000436 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:20.000440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:20.000458 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:45:20.000518 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:20.000522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:20.000582 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:20.000670 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:20.001802 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:20.002520 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:20.003723 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:20.004185 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:20.004276 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:20.008376 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:20.008391 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:20.008408 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:20.008442 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:20.011250 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:20.011896 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:20.011904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:20.011906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:20.012052 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:20.012505 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:20.012512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:20.012519 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:20.012859 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:20.013161 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:20.013437 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 04:45:20.013686 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:45:20.014770 5071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:20.103431 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 4948
I20260619 04:45:20.121510 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 4588
I20260619 04:45:20.122637 4650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:20.234499 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 4588
2026-06-19T04:45:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4025 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:20 dist-test-slave-2qp8 krb5kdc[5123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:20 dist-test-slave-2qp8 krb5kdc[5123](info): set up 2 sockets
Jun 19 04:45:20 dist-test-slave-2qp8 krb5kdc[5123](info): commencing operation
krb5kdc: starting...
W20260619 04:45:21.312098 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:21 dist-test-slave-2qp8 krb5kdc[5123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:21.489879 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:35417
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:35417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:21.799263 5136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:21.799674 5136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:21.799832 5136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:21.807299 5136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:21.807451 5136 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 04:45:21.807528 5136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:21.807619 5136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:21.807718 5136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:21.807809 5136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:21.817345 5136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:35417
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:35417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:21.818682 5136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:21.820022 5136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:21.831899 5142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:21.832067 5141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:21.833106 5144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:21.833343 5136 server_base.cc:1034] running on GCE node
I20260619 04:45:21.834156 5136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:21.835469 5136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:21.836721 5136 hybrid_clock.cc:648] HybridClock initialized: now 1781844321836656 us; error 45 us; skew 500 ppm
I20260619 04:45:21.837056 5136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:21 dist-test-slave-2qp8 krb5kdc[5123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844321, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:21.840267 5136 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:21.841625 5136 webserver.cc:460] Webserver started at http://127.3.243.190:37871/ using document root <none> and password file <none>
I20260619 04:45:21.842316 5136 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:21.842445 5136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:21.842760 5136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:21.845709 5136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "b4259d9ed7a0458fa38ae87749f9d892"
format_stamp: "Formatted at 2026-06-19 04:45:21 on dist-test-slave-2qp8"
server_key: "7be49a8e15e14bf14b7b56bf62541aeb"
server_key_iv: "93d71b7902429be78116fb1e73c33f18"
server_key_version: "encryptionkey@0"
I20260619 04:45:21.846549 5136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "b4259d9ed7a0458fa38ae87749f9d892"
format_stamp: "Formatted at 2026-06-19 04:45:21 on dist-test-slave-2qp8"
server_key: "7be49a8e15e14bf14b7b56bf62541aeb"
server_key_iv: "93d71b7902429be78116fb1e73c33f18"
server_key_version: "encryptionkey@0"
I20260619 04:45:21.851480 5136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:45:21.854885 5151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:21.856261 5136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:21.856436 5136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "b4259d9ed7a0458fa38ae87749f9d892"
format_stamp: "Formatted at 2026-06-19 04:45:21 on dist-test-slave-2qp8"
server_key: "7be49a8e15e14bf14b7b56bf62541aeb"
server_key_iv: "93d71b7902429be78116fb1e73c33f18"
server_key_version: "encryptionkey@0"
I20260619 04:45:21.856652 5136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:21.876722 5136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:21.881112 5136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:21.881388 5136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:21.902365 5136 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:35417
I20260619 04:45:21.902374 5212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:35417 every 8 connection(s)
I20260619 04:45:21.903807 5136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:21.907105 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 5136
I20260619 04:45:21.907325 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:21.907686 5213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:21.908063 4046 external_mini_cluster.cc:1428] Setting key 51ceb0a43fcb61db61517c95487e30c1
I20260619 04:45:21.922012 5213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892: Bootstrap starting.
Jun 19 04:45:21 dist-test-slave-2qp8 krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:21.925863 5213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:21.927232 5213 log.cc:824] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:21.930631 5213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892: No bootstrap required, opened a new log
I20260619 04:45:21.932089 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:21.910038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55478 (local address 127.3.243.190:35417)
0619 04:45:21.910841 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:21.910856 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:21.910921 (+ 65us) server_negotiation.cc:407] Connection header received
0619 04:45:21.912814 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:21.912843 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:21.913368 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:21.913815 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:21.915120 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:21.917190 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:21.918401 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:21.919502 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:21.919691 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:21.923596 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:21.923606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:21.923625 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:21.923678 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:21.927615 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:21.928497 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:21.928503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:21.928509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:21.928725 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:21.929085 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:21.929090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:21.929096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:21.929758 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:21.930087 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:21.930356 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 04:45:21.930780 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":317,"threads_started":1}
I20260619 04:45:21.939265 5213 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4259d9ed7a0458fa38ae87749f9d892" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35417 } }
I20260619 04:45:21.939646 5213 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:21.939739 5213 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4259d9ed7a0458fa38ae87749f9d892, State: Initialized, Role: FOLLOWER
I20260619 04:45:21.940316 5213 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4259d9ed7a0458fa38ae87749f9d892" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35417 } }
I20260619 04:45:21.940593 5213 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:21.940740 5213 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:21.940950 5213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:21.943120 5213 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4259d9ed7a0458fa38ae87749f9d892" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35417 } }
I20260619 04:45:21.943696 5213 leader_election.cc:302] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4259d9ed7a0458fa38ae87749f9d892; no voters:
I20260619 04:45:21.944316 5213 leader_election.cc:288] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:21.944432 5218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:21.945117 5218 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [term 1 LEADER]: Becoming Leader. State: Replica: b4259d9ed7a0458fa38ae87749f9d892, State: Running, Role: LEADER
I20260619 04:45:21.945721 5218 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4259d9ed7a0458fa38ae87749f9d892" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35417 } }
I20260619 04:45:21.946375 5213 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:21.948901 5219 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4259d9ed7a0458fa38ae87749f9d892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4259d9ed7a0458fa38ae87749f9d892" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35417 } } }
I20260619 04:45:21.948894 5220 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4259d9ed7a0458fa38ae87749f9d892. Latest consensus state: current_term: 1 leader_uuid: "b4259d9ed7a0458fa38ae87749f9d892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4259d9ed7a0458fa38ae87749f9d892" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35417 } } }
I20260619 04:45:21.949314 5219 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:21.949342 5220 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:21.950675 5225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:21.955411 5225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:21.964306 5225 catalog_manager.cc:1347] Generated new cluster ID: c62ea6c20bf44eb89e9d618f1741af33
I20260619 04:45:21.964428 5225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:21.981251 5225 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:21.983139 5225 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:21.993502 5225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892: Generated new TSK 0
I20260619 04:45:21.994566 5225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:45:22.048741 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:35417
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:22.374413 5239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:22.374698 5239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:22.374801 5239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:22.382155 5239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:22.382282 5239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:22.382375 5239 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:45:22.382464 5239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:22.392387 5239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.190:35417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:22.393630 5239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:22.395043 5239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:22.408720 5244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:22.408762 5247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:22.408679 5245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:22.409787 5239 server_base.cc:1034] running on GCE node
I20260619 04:45:22.410490 5239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:22.411398 5239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:22.412642 5239 hybrid_clock.cc:648] HybridClock initialized: now 1781844322412567 us; error 56 us; skew 500 ppm
I20260619 04:45:22.412946 5239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:22 dist-test-slave-2qp8 krb5kdc[5123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844322, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:22.416010 5239 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:45:22.417409 5239 webserver.cc:460] Webserver started at http://127.3.243.129:41491/ using document root <none> and password file <none>
I20260619 04:45:22.418107 5239 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:22.418238 5239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:22.418560 5239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:22.421483 5239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "1ed9474500b54f3daae5c2c12c4914a0"
format_stamp: "Formatted at 2026-06-19 04:45:22 on dist-test-slave-2qp8"
server_key: "bdca7b71cc288e9841dd89b7b8da22a3"
server_key_iv: "6d6fde9fedbc01a6fbe7eb825a3b1720"
server_key_version: "encryptionkey@0"
I20260619 04:45:22.422310 5239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "1ed9474500b54f3daae5c2c12c4914a0"
format_stamp: "Formatted at 2026-06-19 04:45:22 on dist-test-slave-2qp8"
server_key: "bdca7b71cc288e9841dd89b7b8da22a3"
server_key_iv: "6d6fde9fedbc01a6fbe7eb825a3b1720"
server_key_version: "encryptionkey@0"
I20260619 04:45:22.427651 5239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:45:22.431222 5254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:22.432678 5239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:22.432861 5239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "1ed9474500b54f3daae5c2c12c4914a0"
format_stamp: "Formatted at 2026-06-19 04:45:22 on dist-test-slave-2qp8"
server_key: "bdca7b71cc288e9841dd89b7b8da22a3"
server_key_iv: "6d6fde9fedbc01a6fbe7eb825a3b1720"
server_key_version: "encryptionkey@0"
I20260619 04:45:22.433101 5239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:22.456614 5239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:22.460668 5239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:22.460963 5239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:22.463490 5239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:22.463599 5239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:22.463763 5239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:22.463840 5239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:45:22 dist-test-slave-2qp8 krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844322, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:22.488632 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:22.464280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:53267 (local address 127.3.243.190:35417)
0619 04:45:22.464789 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:22.464799 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:22.466590 (+ 1791us) server_negotiation.cc:407] Connection header received
0619 04:45:22.468710 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:22.468715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:22.468815 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:22.468934 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:22.471222 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.472565 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.475136 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.475804 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.475940 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:22.481391 (+ 5451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:22.481399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:22.481407 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:22.481461 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:22.485016 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:22.486089 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:22.486096 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:22.486101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:22.486286 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:22.486818 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:22.486824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:22.486828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:22.487118 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:22.487283 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:22.487959 (+ 676us) server_negotiation.cc:299] Negotiation successful
0619 04:45:22.488243 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":239,"threads_started":1}
I20260619 04:45:22.490082 5264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:22.465627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35417 (local address 127.3.243.129:53267)
0619 04:45:22.466423 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:22.466461 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:22.468417 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:22.469216 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:22.469232 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:22.470105 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:22.471001 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:22.471036 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.472773 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.472784 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:22.474972 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:22.474984 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.476055 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.476062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:22.476395 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:22.477245 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:22.477288 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:22.481089 (+ 3801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:22.485162 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:22.485172 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:22.485190 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:22.485944 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:22.486430 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:22.486435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:22.486441 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:22.486693 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:22.487258 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:22.487266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:22.487620 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 04:45:22.487885 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 04:45:22.488512 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":327,"threads_started":1}
I20260619 04:45:22.510021 5239 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:40805
I20260619 04:45:22.510033 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:40805 every 8 connection(s)
I20260619 04:45:22.511209 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:22.496622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35417 (local address 127.3.243.129:56259)
0619 04:45:22.497330 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:22.497354 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:22.497537 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:22.497933 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:22.497938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:22.498400 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:22.498682 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:22.498690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.499968 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.499973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:22.501453 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:22.501465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.502400 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.502407 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:22.502593 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:22.503341 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:22.503358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:22.504640 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:22.508411 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:22.508416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:22.508420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:22.509030 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:22.509549 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:22.509554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:22.509559 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:22.509756 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:22.510377 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:22.510380 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:22.510531 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 04:45:22.510637 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 04:45:22.510880 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":402,"threads_started":1}
I20260619 04:45:22.511359 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:22.496273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:56259 (local address 127.3.243.190:35417)
0619 04:45:22.496490 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:22.496498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:22.497466 (+ 968us) server_negotiation.cc:407] Connection header received
0619 04:45:22.497735 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:22.497740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:22.497801 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:22.497903 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:22.498823 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.499832 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.501605 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.502185 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.502319 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:22.504837 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:22.504861 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:22.504867 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:22.504936 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:22.508290 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:22.509221 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:22.509226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:22.509229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:22.509413 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:22.509896 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:22.509900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:22.509903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:22.510251 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:22.510403 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:22.510793 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 04:45:22.511016 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 04:45:22.511606 5239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:22.513588 5375 heartbeater.cc:347] Connected to a master server at 127.3.243.190:35417
I20260619 04:45:22.514403 5375 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:22.515025 5375 heartbeater.cc:511] Master 127.3.243.190:35417 requested a full tablet report, sending...
I20260619 04:45:22.516597 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 5239
I20260619 04:45:22.516814 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:22.517139 5168 ts_manager.cc:194] Registered new tserver with Master: 1ed9474500b54f3daae5c2c12c4914a0 (127.3.243.129:40805)
I20260619 04:45:22.517622 4046 external_mini_cluster.cc:1428] Setting key 97e0515be602a4b26bf7a39d92f00889
I20260619 04:45:22.519049 5168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:53267
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:45:22.557036 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:35417
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:22.871634 5379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:22.872021 5379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:22.872236 5379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:22.879683 5379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:22.879868 5379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:22.880033 5379 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:45:22.880204 5379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:22.890225 5379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.190:35417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:22.891552 5379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:22.893131 5379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:22.906553 5385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:22.906633 5384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:22.906633 5387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:22.908058 5379 server_base.cc:1034] running on GCE node
I20260619 04:45:22.908838 5379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:22.909932 5379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:22.911196 5379 hybrid_clock.cc:648] HybridClock initialized: now 1781844322911118 us; error 44 us; skew 500 ppm
I20260619 04:45:22.911554 5379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:22 dist-test-slave-2qp8 krb5kdc[5123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844322, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:22.922667 5379 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:45:22.924278 5379 webserver.cc:460] Webserver started at http://127.3.243.130:32883/ using document root <none> and password file <none>
I20260619 04:45:22.924929 5379 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:22.925021 5379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:22.925271 5379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:22.928251 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "f17ce4cba4ae4763a6c6018ca5a0b406"
format_stamp: "Formatted at 2026-06-19 04:45:22 on dist-test-slave-2qp8"
server_key: "0e5f21873dd1ddf54654a5df71a87fe2"
server_key_iv: "9603b37df665a47bb30e90e1c1c952b6"
server_key_version: "encryptionkey@0"
I20260619 04:45:22.929039 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "f17ce4cba4ae4763a6c6018ca5a0b406"
format_stamp: "Formatted at 2026-06-19 04:45:22 on dist-test-slave-2qp8"
server_key: "0e5f21873dd1ddf54654a5df71a87fe2"
server_key_iv: "9603b37df665a47bb30e90e1c1c952b6"
server_key_version: "encryptionkey@0"
I20260619 04:45:22.933768 5379 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:45:22.937029 5394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:22.938402 5379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:22.938582 5379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "f17ce4cba4ae4763a6c6018ca5a0b406"
format_stamp: "Formatted at 2026-06-19 04:45:22 on dist-test-slave-2qp8"
server_key: "0e5f21873dd1ddf54654a5df71a87fe2"
server_key_iv: "9603b37df665a47bb30e90e1c1c952b6"
server_key_version: "encryptionkey@0"
I20260619 04:45:22.938794 5379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:22.956790 5379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:22.961135 5379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:22.961406 5379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:22.964046 5379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:22.964183 5379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 04:45:22.964361 5379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:22.964435 5379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:45:22 dist-test-slave-2qp8 krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844322, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:22.988024 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:22.964411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:47669 (local address 127.3.243.190:35417)
0619 04:45:22.964626 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:22.964632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:22.966135 (+ 1503us) server_negotiation.cc:407] Connection header received
0619 04:45:22.968420 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:22.968426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:22.968536 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:22.968646 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:22.971374 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.972408 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.975516 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.976412 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.976556 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:22.980335 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:22.980340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:22.980343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:22.980376 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:22.984379 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:22.985416 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:22.985421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:22.985425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:22.985578 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:22.986175 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:22.986180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:22.986183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:22.986462 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:22.986574 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:22.987637 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0619 04:45:22.987793 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 04:45:22.990355 5403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:22.965101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35417 (local address 127.3.243.130:47669)
0619 04:45:22.965949 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:22.966004 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:22.968052 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:22.968937 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:22.968950 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:22.969985 (+ 1035us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:22.971149 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:22.971189 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.972592 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.972601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:22.975299 (+ 2698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:22.975320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:22.976635 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:22.976640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:22.976811 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:22.977545 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:22.977566 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:22.980039 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:22.984545 (+ 4506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:22.984559 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:22.984579 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:22.985230 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:22.985716 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:22.985722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:22.985726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:22.986038 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:22.986608 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:22.986619 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:22.987203 (+ 584us) client_negotiation.cc:769] Sending connection context
0619 04:45:22.987565 (+ 362us) client_negotiation.cc:240] Negotiation successful
0619 04:45:22.988290 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":381,"threads_started":1}
I20260619 04:45:23.015347 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:22.999259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:41285 (local address 127.3.243.190:35417)
0619 04:45:22.999459 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:22.999467 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.000381 (+ 914us) server_negotiation.cc:407] Connection header received
0619 04:45:23.000616 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.000624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.000733 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.000850 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:23.001758 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.003071 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.004729 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.005253 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.005391 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.008176 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:23.008184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:23.008189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:23.008228 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:23.011986 (+ 3758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:23.012975 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:23.012983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:23.012986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:23.013170 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:23.013622 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:23.013628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:23.013631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:23.014207 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:23.014366 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.014773 (+ 407us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.014996 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:45:23.015506 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:22.999368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35417 (local address 127.3.243.130:41285)
0619 04:45:23.000236 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:23.000273 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:23.000455 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:23.000885 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:23.000891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:23.001322 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:23.001602 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.001611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.003252 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.003256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.004579 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.004592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.005643 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.005648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.005889 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.006642 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:23.006676 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:23.007866 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:23.012188 (+ 4322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:23.012196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:23.012207 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:23.012834 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:23.013331 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:23.013337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:23.013342 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:23.013500 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:23.014335 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:23.014340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:23.014533 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 04:45:23.014671 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 04:45:23.015011 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":486,"threads_started":1}
I20260619 04:45:23.018910 5379 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:37689
I20260619 04:45:23.018936 5513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:37689 every 8 connection(s)
I20260619 04:45:23.020247 5379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:23.022225 5514 heartbeater.cc:347] Connected to a master server at 127.3.243.190:35417
I20260619 04:45:23.022421 5514 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:23.022997 5514 heartbeater.cc:511] Master 127.3.243.190:35417 requested a full tablet report, sending...
I20260619 04:45:23.024367 5168 ts_manager.cc:194] Registered new tserver with Master: f17ce4cba4ae4763a6c6018ca5a0b406 (127.3.243.130:37689)
I20260619 04:45:23.024386 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 5379
I20260619 04:45:23.024592 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:23.025032 4046 external_mini_cluster.cc:1428] Setting key 24750bad17fbf7df6c7e8ff55b8255c8
I20260619 04:45:23.025261 5168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:47669
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:45:23.063843 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:35417
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:23.375032 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:23.375339 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:23.375505 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:23.382827 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:23.382997 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:23.383144 5518 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:45:23.383285 5518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:45:23.393777 5518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.190:35417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:23.395032 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:23.396447 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:23.409461 5523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:23.409541 5526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:23.409426 5524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:23.410787 5518 server_base.cc:1034] running on GCE node
I20260619 04:45:23.411469 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:23.412446 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:23.413661 5518 hybrid_clock.cc:648] HybridClock initialized: now 1781844323413606 us; error 31 us; skew 500 ppm
I20260619 04:45:23.413928 5518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:23 dist-test-slave-2qp8 krb5kdc[5123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844323, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:23.416977 5518 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:45:23.418311 5518 webserver.cc:460] Webserver started at http://127.3.243.131:37405/ using document root <none> and password file <none>
I20260619 04:45:23.419018 5518 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:23.419103 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:23.419368 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:23.422293 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "99c906bb77a14d15838953a73feca757"
format_stamp: "Formatted at 2026-06-19 04:45:23 on dist-test-slave-2qp8"
server_key: "6ee0c4dd8463b4f6fb3f41a4609414dc"
server_key_iv: "32319b2b2c9731687239c33389dea0f2"
server_key_version: "encryptionkey@0"
I20260619 04:45:23.423058 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "99c906bb77a14d15838953a73feca757"
format_stamp: "Formatted at 2026-06-19 04:45:23 on dist-test-slave-2qp8"
server_key: "6ee0c4dd8463b4f6fb3f41a4609414dc"
server_key_iv: "32319b2b2c9731687239c33389dea0f2"
server_key_version: "encryptionkey@0"
I20260619 04:45:23.428054 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:45:23.431653 5533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:23.433158 5518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:23.433300 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "99c906bb77a14d15838953a73feca757"
format_stamp: "Formatted at 2026-06-19 04:45:23 on dist-test-slave-2qp8"
server_key: "6ee0c4dd8463b4f6fb3f41a4609414dc"
server_key_iv: "32319b2b2c9731687239c33389dea0f2"
server_key_version: "encryptionkey@0"
I20260619 04:45:23.433480 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:23.469261 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:23.473299 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:23.473519 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:23.476100 5518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:23.476256 5518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 04:45:23.476410 5518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:23.476486 5518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:45:23 dist-test-slave-2qp8 krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844323, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:23.499148 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.476804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:60199 (local address 127.3.243.190:35417)
0619 04:45:23.477018 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:23.477024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.478384 (+ 1360us) server_negotiation.cc:407] Connection header received
0619 04:45:23.480601 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.480606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.480696 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.480804 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:23.482997 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.484368 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.486824 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.487451 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.487588 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.492367 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:23.492375 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:23.492381 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:23.492422 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:23.495913 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:23.496850 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:23.496856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:23.496860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:23.496989 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:23.497419 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:23.497425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:23.497428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:23.497700 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:23.497854 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.498584 (+ 730us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.498804 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:45:23.500080 5542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.477368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35417 (local address 127.3.243.131:60199)
0619 04:45:23.478205 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:23.478249 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:23.480321 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:23.481076 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:23.481091 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:23.481895 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:23.482715 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.482733 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.484582 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.484590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.486655 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.486667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.487698 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.487709 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.487931 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.488795 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:23.488821 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:23.491966 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:23.496059 (+ 4093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:23.496068 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:23.496084 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:23.496676 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:23.497099 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:23.497108 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:23.497112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:23.497301 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:23.497826 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:23.497833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:23.498177 (+ 344us) client_negotiation.cc:769] Sending connection context
0619 04:45:23.498426 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 04:45:23.498907 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":354,"threads_started":1}
I20260619 04:45:23.522434 5375 heartbeater.cc:503] Master 127.3.243.190:35417 was elected leader, sending a full tablet report...
I20260619 04:45:23.523267 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.505900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35417 (local address 127.3.243.131:58575)
0619 04:45:23.506977 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:23.507002 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:23.507176 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:23.507526 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:23.507531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:23.508212 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:23.508683 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.508697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.509901 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.509906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.512077 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.512098 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.513818 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.513830 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.514074 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.514859 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:23.514890 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:23.516251 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:23.520286 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:23.520290 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:23.520292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:23.520860 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:23.521297 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:23.521307 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:23.521310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:23.521436 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:23.522143 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:23.522146 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:23.522361 (+ 215us) client_negotiation.cc:769] Sending connection context
0619 04:45:23.522535 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 04:45:23.522838 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"thread_start_us":702,"threads_started":1}
I20260619 04:45:23.523492 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.505888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:58575 (local address 127.3.243.190:35417)
0619 04:45:23.506096 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:23.506104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.507103 (+ 999us) server_negotiation.cc:407] Connection header received
0619 04:45:23.507309 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.507312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.507381 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.507510 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:23.508904 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.509759 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.512309 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.513120 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.513242 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.516479 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:23.516486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:23.516493 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:23.516529 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:23.520170 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:23.520994 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:23.521001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:23.521007 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:23.521178 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:23.521568 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:23.521576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:23.521582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:23.522030 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:23.522208 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.522694 (+ 486us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.522983 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:45:23.532069 5518 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:38719
I20260619 04:45:23.532161 5652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:38719 every 8 connection(s)
I20260619 04:45:23.533255 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:45:23.535223 5653 heartbeater.cc:347] Connected to a master server at 127.3.243.190:35417
I20260619 04:45:23.535442 5653 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:23.536059 5653 heartbeater.cc:511] Master 127.3.243.190:35417 requested a full tablet report, sending...
I20260619 04:45:23.537454 5168 ts_manager.cc:194] Registered new tserver with Master: 99c906bb77a14d15838953a73feca757 (127.3.243.131:38719)
I20260619 04:45:23.538455 5168 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:60199
I20260619 04:45:23.541141 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 5518
I20260619 04:45:23.541296 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:45:23.541747 4046 external_mini_cluster.cc:1428] Setting key 44caeef7ae499edcd1156b8e4abe3ef6
I20260619 04:45:23.544517 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 04:45:23 dist-test-slave-2qp8 krb5kdc[5123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:45:23 dist-test-slave-2qp8 krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:23.588047 5263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.575026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.3.243.190:35417)
0619 04:45:23.575250 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:23.575258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.575409 (+ 151us) server_negotiation.cc:407] Connection header received
0619 04:45:23.575617 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.575620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.575686 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.575774 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:23.577129 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.577856 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.579040 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.579444 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.579536 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.583211 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:23.583215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:23.583218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:23.583251 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:23.585858 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:23.586399 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:23.586402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:23.586404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:23.586518 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:23.586809 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:23.586812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:23.586814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:23.587074 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:23.587260 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.587503 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.587658 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 04:45:23.591917 5168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:45:23.598902 5168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:23.634023 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.611498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.3.243.130:37689)
0619 04:45:23.625013 (+ 13515us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:23.625022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.625983 (+ 961us) server_negotiation.cc:407] Connection header received
0619 04:45:23.626111 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.626118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.626383 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.626605 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:23.627371 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.628900 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.631622 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.632939 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.633056 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.633235 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.633590 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.633744 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13339,"thread_start_us":13420,"threads_started":1}
I20260619 04:45:23.634102 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.610489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:40805 (local address 127.0.0.1:45762)
0619 04:45:23.622664 (+ 12175us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:23.622686 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:23.622831 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:23.626126 (+ 3295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:23.626134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:23.626168 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:23.626558 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.626574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.628521 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.628526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.630969 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.630983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.632930 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.632938 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.633115 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.633161 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 04:45:23.633305 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 04:45:23.633518 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12026,"mutex_wait_us":5,"thread_start_us":11873,"threads_started":1}
I20260619 04:45:23.634172 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.610442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:38719 (local address 127.0.0.1:42890)
0619 04:45:23.622069 (+ 11627us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:23.622118 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:23.622357 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:23.626773 (+ 4416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:23.626779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:23.626806 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:23.627233 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.627247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.629389 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.629395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.631695 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.631706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.633364 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.633369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.633547 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.633575 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:45:23.633663 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 04:45:23.633779 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11271,"mutex_wait_us":3,"thread_start_us":11102,"threads_started":1}
I20260619 04:45:23.634203 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.611052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.3.243.129:40805)
0619 04:45:23.625429 (+ 14377us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:23.625440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.625471 (+ 31us) server_negotiation.cc:407] Connection header received
0619 04:45:23.625595 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.625605 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.625896 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.626772 (+ 876us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:23.626933 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.628400 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.631172 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.632729 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.632869 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.633098 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.633755 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.633929 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14196,"thread_start_us":14073,"threads_started":1}
I20260619 04:45:23.634773 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.610596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.3.243.131:38719)
0619 04:45:23.626247 (+ 15651us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:23.626253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.626277 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:45:23.626347 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.626352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.626562 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.627429 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:23.627735 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.629253 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.631845 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.632856 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.632971 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.633217 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.634380 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.634486 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15506,"thread_start_us":15325,"threads_started":1}
I20260619 04:45:23.635362 5669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.610852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:37689 (local address 127.0.0.1:35214)
0619 04:45:23.625723 (+ 14871us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:23.625740 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:23.625851 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:23.626598 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:23.626605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:23.626633 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:23.627162 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.627175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.629047 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.629052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.631435 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.631448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.633250 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.633259 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.633391 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.633413 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 04:45:23.633525 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 04:45:23.633651 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14751,"mutex_wait_us":27,"spinlock_wait_cycles":1920,"thread_start_us":14622,"threads_started":1}
I20260619 04:45:23.638447 5583 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab93e3bfd844e038fbdf229829927cd (DEFAULT_TABLE table=test-table [id=325207ef93d14198b305ec3e1ea15862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:23.638447 5305 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab93e3bfd844e038fbdf229829927cd (DEFAULT_TABLE table=test-table [id=325207ef93d14198b305ec3e1ea15862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:23.638700 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 9ab93e3bfd844e038fbdf229829927cd (DEFAULT_TABLE table=test-table [id=325207ef93d14198b305ec3e1ea15862]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:23.639868 5305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab93e3bfd844e038fbdf229829927cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:23.639881 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab93e3bfd844e038fbdf229829927cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:23.639890 5444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ab93e3bfd844e038fbdf229829927cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:23.651229 5672 tablet_bootstrap.cc:492] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: Bootstrap starting.
I20260619 04:45:23.655314 5674 tablet_bootstrap.cc:492] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: Bootstrap starting.
I20260619 04:45:23.655589 5672 tablet_bootstrap.cc:654] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:23.656940 5672 log.cc:824] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:23.657017 5673 tablet_bootstrap.cc:492] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: Bootstrap starting.
I20260619 04:45:23.658700 5674 tablet_bootstrap.cc:654] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:23.659969 5674 log.cc:824] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:23.660254 5673 tablet_bootstrap.cc:654] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:23.660529 5672 tablet_bootstrap.cc:492] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: No bootstrap required, opened a new log
I20260619 04:45:23.660866 5672 ts_tablet_manager.cc:1397] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260619 04:45:23.661438 5673 log.cc:824] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:23.663486 5674 tablet_bootstrap.cc:492] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: No bootstrap required, opened a new log
I20260619 04:45:23.663743 5674 ts_tablet_manager.cc:1397] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 04:45:23.669998 5673 tablet_bootstrap.cc:492] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: No bootstrap required, opened a new log
I20260619 04:45:23.670282 5673 ts_tablet_manager.cc:1397] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260619 04:45:23.671355 5672 raft_consensus.cc:348] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } }
I20260619 04:45:23.671921 5672 raft_consensus.cc:374] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:23.672062 5672 raft_consensus.cc:729] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17ce4cba4ae4763a6c6018ca5a0b406, State: Initialized, Role: FOLLOWER
I20260619 04:45:23.672842 5672 consensus_queue.cc:260] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } }
I20260619 04:45:23.673872 5514 heartbeater.cc:503] Master 127.3.243.190:35417 was elected leader, sending a full tablet report...
I20260619 04:45:23.673974 5674 raft_consensus.cc:348] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } }
I20260619 04:45:23.674379 5674 raft_consensus.cc:374] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:23.674472 5674 raft_consensus.cc:729] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed9474500b54f3daae5c2c12c4914a0, State: Initialized, Role: FOLLOWER
I20260619 04:45:23.675019 5674 consensus_queue.cc:260] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } }
I20260619 04:45:23.675877 5672 ts_tablet_manager.cc:1428] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260619 04:45:23.677601 5674 ts_tablet_manager.cc:1428] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260619 04:45:23.681088 5673 raft_consensus.cc:348] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } }
I20260619 04:45:23.681427 5673 raft_consensus.cc:374] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:23.681569 5673 raft_consensus.cc:729] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c906bb77a14d15838953a73feca757, State: Initialized, Role: FOLLOWER
I20260619 04:45:23.682171 5673 consensus_queue.cc:260] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } }
I20260619 04:45:23.683121 5653 heartbeater.cc:503] Master 127.3.243.190:35417 was elected leader, sending a full tablet report...
I20260619 04:45:23.683749 5673 ts_tablet_manager.cc:1428] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
W20260619 04:45:23.764159 5376 tablet.cc:2369] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:45:23.772400 5515 tablet.cc:2369] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:45:23.784762 5654 tablet.cc:2369] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:45:23.785619 5679 raft_consensus.cc:482] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:23.785887 5679 raft_consensus.cc:504] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } }
I20260619 04:45:23.787371 5679 leader_election.cc:288] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99c906bb77a14d15838953a73feca757 (127.3.243.131:38719), f17ce4cba4ae4763a6c6018ca5a0b406 (127.3.243.130:37689)
I20260619 04:45:23.796603 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.787767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:40383 (local address 127.3.243.131:38719)
0619 04:45:23.787962 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:23.787968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.788727 (+ 759us) server_negotiation.cc:407] Connection header received
0619 04:45:23.788938 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.788944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.789030 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.789115 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:23.789925 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.791068 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.794236 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.795458 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.795663 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.795732 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.796213 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.796343 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:45:23.796623 5671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.787864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:59407 (local address 127.3.243.130:37689)
0619 04:45:23.788054 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:23.788062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.788726 (+ 664us) server_negotiation.cc:407] Connection header received
0619 04:45:23.788938 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.788944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.789020 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.789115 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:23.789925 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.791071 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.794242 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.795461 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.795661 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.795733 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.796214 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.796343 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:45:23.796936 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.787694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:38719 (local address 127.3.243.129:40383)
0619 04:45:23.788589 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:23.788614 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:23.788804 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:23.789149 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:23.789155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:23.789194 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:23.789773 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.789785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.791206 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.791210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.794072 (+ 2862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.794087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.795753 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.795758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.796017 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.796068 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 04:45:23.796179 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 04:45:23.796383 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":339,"spinlock_wait_cycles":48768,"thread_start_us":503,"threads_started":1}
I20260619 04:45:23.796957 5682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.787800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:37689 (local address 127.3.243.129:59407)
0619 04:45:23.788616 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:23.788630 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:23.788768 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:23.789149 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:23.789155 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:23.789184 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:23.789772 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.789785 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.791204 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.791210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.794073 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:23.794087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.795727 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.795737 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:23.796011 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.796071 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 04:45:23.796177 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 04:45:23.796382 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":278,"thread_start_us":479,"threads_started":1}
I20260619 04:45:23.798032 5608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab93e3bfd844e038fbdf229829927cd" candidate_uuid: "1ed9474500b54f3daae5c2c12c4914a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c906bb77a14d15838953a73feca757" is_pre_election: true
I20260619 04:45:23.798152 5469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab93e3bfd844e038fbdf229829927cd" candidate_uuid: "1ed9474500b54f3daae5c2c12c4914a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" is_pre_election: true
I20260619 04:45:23.798568 5608 raft_consensus.cc:2454] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed9474500b54f3daae5c2c12c4914a0 in term 0.
I20260619 04:45:23.798641 5469 raft_consensus.cc:2454] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed9474500b54f3daae5c2c12c4914a0 in term 0.
I20260619 04:45:23.799293 5258 leader_election.cc:302] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ed9474500b54f3daae5c2c12c4914a0, 99c906bb77a14d15838953a73feca757, f17ce4cba4ae4763a6c6018ca5a0b406; no voters:
I20260619 04:45:23.799698 5679 raft_consensus.cc:2781] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:23.799849 5679 raft_consensus.cc:482] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:23.799952 5679 raft_consensus.cc:3037] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:23.802277 5679 raft_consensus.cc:504] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } }
I20260619 04:45:23.802820 5679 leader_election.cc:288] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [CANDIDATE]: Term 1 election: Requested vote from peers 99c906bb77a14d15838953a73feca757 (127.3.243.131:38719), f17ce4cba4ae4763a6c6018ca5a0b406 (127.3.243.130:37689)
I20260619 04:45:23.803256 5469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab93e3bfd844e038fbdf229829927cd" candidate_uuid: "1ed9474500b54f3daae5c2c12c4914a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406"
I20260619 04:45:23.803259 5608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ab93e3bfd844e038fbdf229829927cd" candidate_uuid: "1ed9474500b54f3daae5c2c12c4914a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c906bb77a14d15838953a73feca757"
I20260619 04:45:23.803463 5469 raft_consensus.cc:3037] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:23.803464 5608 raft_consensus.cc:3037] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:23.806624 5469 raft_consensus.cc:2454] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ed9474500b54f3daae5c2c12c4914a0 in term 1.
I20260619 04:45:23.806629 5608 raft_consensus.cc:2454] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ed9474500b54f3daae5c2c12c4914a0 in term 1.
I20260619 04:45:23.807066 5257 leader_election.cc:302] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1ed9474500b54f3daae5c2c12c4914a0, 99c906bb77a14d15838953a73feca757, f17ce4cba4ae4763a6c6018ca5a0b406; no voters:
I20260619 04:45:23.807420 5679 raft_consensus.cc:2781] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:23.807847 5679 raft_consensus.cc:686] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed9474500b54f3daae5c2c12c4914a0, State: Running, Role: LEADER
I20260619 04:45:23.808364 5679 consensus_queue.cc:237] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } }
I20260619 04:45:23.812856 5167 catalog_manager.cc:5495] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ed9474500b54f3daae5c2c12c4914a0 (127.3.243.129). New cstate: current_term: 1 leader_uuid: "1ed9474500b54f3daae5c2c12c4914a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20260619 04:45:23.875900 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:23.870635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.3.243.129:40805)
0619 04:45:23.870833 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:23.870837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:23.870862 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:45:23.871134 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:23.871138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:23.871235 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:23.871410 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:23.871858 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.872710 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.874044 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:23.874409 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:23.874506 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:23.874918 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:23.875413 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:23.875527 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:23.875630 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 04:45:23.875695 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:45:23.883836 5469 raft_consensus.cc:1261] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Refusing update from remote peer 1ed9474500b54f3daae5c2c12c4914a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:23.883836 5608 raft_consensus.cc:1261] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Refusing update from remote peer 1ed9474500b54f3daae5c2c12c4914a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:23.884891 5679 consensus_queue.cc:1035] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:23.885181 5683 consensus_queue.cc:1035] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:23.899960 5686 mvcc.cc:205] Tried to move back new op lower bound from 7298434350617939968 to 7298434350322348032. Current Snapshot: MvccSnapshot[applied={T|T < 7298434350617939968 or (T in {7298434350617939968})}]
I20260619 04:45:23.915874 5168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:23.916224 5168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:23.920415 5168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892: Sending DeleteTablet for 3 replicas of tablet 9ab93e3bfd844e038fbdf229829927cd
I20260619 04:45:23.921700 5583 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab93e3bfd844e038fbdf229829927cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:23 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:42890
I20260619 04:45:23.921676 5305 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab93e3bfd844e038fbdf229829927cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:23 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:45762
I20260619 04:45:23.921999 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ab93e3bfd844e038fbdf229829927cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:23 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:35214
I20260619 04:45:23.924057 5168 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:45:23.925163 5168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:23.936391 5695 tablet_replica.cc:330] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: stopping tablet replica
I20260619 04:45:23.936574 5582 tablet_service.cc:1467] Processing CreateTablet for tablet c99c2846220e47139106e2c1852ce72c (DEFAULT_TABLE table=test-table [id=37a5d4a1f2214a1c954e0d8981c167ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:23.937116 5696 tablet_replica.cc:330] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: stopping tablet replica
I20260619 04:45:23.937177 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99c2846220e47139106e2c1852ce72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:23.937711 5443 tablet_service.cc:1467] Processing CreateTablet for tablet c99c2846220e47139106e2c1852ce72c (DEFAULT_TABLE table=test-table [id=37a5d4a1f2214a1c954e0d8981c167ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:23.938398 5443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99c2846220e47139106e2c1852ce72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:23.945072 5673 tablet_bootstrap.cc:492] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: Bootstrap starting.
I20260619 04:45:23.946759 5696 raft_consensus.cc:2229] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:23.946986 5672 tablet_bootstrap.cc:492] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: Bootstrap starting.
I20260619 04:45:23.936271 5304 tablet_service.cc:1467] Processing CreateTablet for tablet c99c2846220e47139106e2c1852ce72c (DEFAULT_TABLE table=test-table [id=37a5d4a1f2214a1c954e0d8981c167ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:23.947472 5695 raft_consensus.cc:2229] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:45:23.946816 5673 tablet_bootstrap.cc:654] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:23.948699 5672 tablet_bootstrap.cc:654] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:23.949222 5696 raft_consensus.cc:2258] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:23.947510 5304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99c2846220e47139106e2c1852ce72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:23.950634 5697 tablet_replica.cc:330] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: stopping tablet replica
I20260619 04:45:23.953608 5673 tablet_bootstrap.cc:492] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: No bootstrap required, opened a new log
I20260619 04:45:23.953768 5673 ts_tablet_manager.cc:1397] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260619 04:45:23.953876 5696 ts_tablet_manager.cc:1905] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:23.954587 5673 raft_consensus.cc:348] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:23.954807 5673 raft_consensus.cc:374] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:23.954921 5673 raft_consensus.cc:729] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c906bb77a14d15838953a73feca757, State: Initialized, Role: FOLLOWER
I20260619 04:45:23.955219 5673 consensus_queue.cc:260] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:23.956722 5673 ts_tablet_manager.cc:1428] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 04:45:23.958686 5674 tablet_bootstrap.cc:492] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: Bootstrap starting.
I20260619 04:45:23.959126 5695 raft_consensus.cc:2258] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:23.960770 5674 tablet_bootstrap.cc:654] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:23.954669 5697 raft_consensus.cc:2229] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:23.961747 5697 raft_consensus.cc:2258] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:23.969568 5695 ts_tablet_manager.cc:1905] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:23.969568 5697 ts_tablet_manager.cc:1905] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:23.974139 5696 ts_tablet_manager.cc:1918] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:45:23.974424 5674 tablet_bootstrap.cc:492] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: No bootstrap required, opened a new log
I20260619 04:45:23.974587 5674 ts_tablet_manager.cc:1397] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260619 04:45:23.975526 5674 raft_consensus.cc:348] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:23.975812 5674 raft_consensus.cc:374] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:23.975858 5672 tablet_bootstrap.cc:492] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: No bootstrap required, opened a new log
I20260619 04:45:23.975940 5674 raft_consensus.cc:729] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed9474500b54f3daae5c2c12c4914a0, State: Initialized, Role: FOLLOWER
I20260619 04:45:23.976182 5672 ts_tablet_manager.cc:1397] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.016s
I20260619 04:45:23.976277 5674 consensus_queue.cc:260] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:23.977067 5672 raft_consensus.cc:348] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:23.977357 5672 raft_consensus.cc:374] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:23.977433 5695 ts_tablet_manager.cc:1918] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:45:23.977550 5672 raft_consensus.cc:729] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17ce4cba4ae4763a6c6018ca5a0b406, State: Initialized, Role: FOLLOWER
I20260619 04:45:23.977595 5695 log.cc:1196] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/9ab93e3bfd844e038fbdf229829927cd
I20260619 04:45:23.978276 5695 ts_tablet_manager.cc:1939] T 9ab93e3bfd844e038fbdf229829927cd P 1ed9474500b54f3daae5c2c12c4914a0: Deleting consensus metadata
I20260619 04:45:23.978564 5672 consensus_queue.cc:260] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:23.979225 5674 ts_tablet_manager.cc:1428] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260619 04:45:23.979954 5672 ts_tablet_manager.cc:1428] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 04:45:23.981570 5696 log.cc:1196] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/9ab93e3bfd844e038fbdf229829927cd
I20260619 04:45:23.982142 5696 ts_tablet_manager.cc:1939] T 9ab93e3bfd844e038fbdf229829927cd P 99c906bb77a14d15838953a73feca757: Deleting consensus metadata
I20260619 04:45:23.983912 5152 catalog_manager.cc:4841] TS 1ed9474500b54f3daae5c2c12c4914a0 (127.3.243.129:40805): tablet 9ab93e3bfd844e038fbdf229829927cd (table test-table [id=325207ef93d14198b305ec3e1ea15862]) successfully deleted
I20260619 04:45:23.984030 5154 catalog_manager.cc:4841] TS 99c906bb77a14d15838953a73feca757 (127.3.243.131:38719): tablet 9ab93e3bfd844e038fbdf229829927cd (table test-table [id=325207ef93d14198b305ec3e1ea15862]) successfully deleted
I20260619 04:45:23.984937 5697 ts_tablet_manager.cc:1918] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:45:23.985109 5697 log.cc:1196] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/9ab93e3bfd844e038fbdf229829927cd
I20260619 04:45:23.985546 5697 ts_tablet_manager.cc:1939] T 9ab93e3bfd844e038fbdf229829927cd P f17ce4cba4ae4763a6c6018ca5a0b406: Deleting consensus metadata
I20260619 04:45:23.986966 5155 catalog_manager.cc:4841] TS f17ce4cba4ae4763a6c6018ca5a0b406 (127.3.243.130:37689): tablet 9ab93e3bfd844e038fbdf229829927cd (table test-table [id=325207ef93d14198b305ec3e1ea15862]) successfully deleted
I20260619 04:45:24.035454 5683 raft_consensus.cc:482] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:24.035697 5683 raft_consensus.cc:504] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.036506 5683 leader_election.cc:288] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f17ce4cba4ae4763a6c6018ca5a0b406 (127.3.243.130:37689), 99c906bb77a14d15838953a73feca757 (127.3.243.131:38719)
I20260619 04:45:24.037020 5469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99c2846220e47139106e2c1852ce72c" candidate_uuid: "1ed9474500b54f3daae5c2c12c4914a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" is_pre_election: true
I20260619 04:45:24.037199 5608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99c2846220e47139106e2c1852ce72c" candidate_uuid: "1ed9474500b54f3daae5c2c12c4914a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c906bb77a14d15838953a73feca757" is_pre_election: true
I20260619 04:45:24.037307 5469 raft_consensus.cc:2454] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed9474500b54f3daae5c2c12c4914a0 in term 0.
I20260619 04:45:24.037437 5608 raft_consensus.cc:2454] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ed9474500b54f3daae5c2c12c4914a0 in term 0.
I20260619 04:45:24.037945 5257 leader_election.cc:302] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed9474500b54f3daae5c2c12c4914a0, 99c906bb77a14d15838953a73feca757; no voters:
I20260619 04:45:24.038257 5683 raft_consensus.cc:2781] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:24.038362 5683 raft_consensus.cc:482] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:24.038480 5683 raft_consensus.cc:3037] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:24.040784 5683 raft_consensus.cc:504] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.041767 5469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99c2846220e47139106e2c1852ce72c" candidate_uuid: "1ed9474500b54f3daae5c2c12c4914a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406"
I20260619 04:45:24.041777 5608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99c2846220e47139106e2c1852ce72c" candidate_uuid: "1ed9474500b54f3daae5c2c12c4914a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c906bb77a14d15838953a73feca757"
I20260619 04:45:24.041978 5469 raft_consensus.cc:3037] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:24.041977 5608 raft_consensus.cc:3037] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:24.045064 5608 raft_consensus.cc:2454] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ed9474500b54f3daae5c2c12c4914a0 in term 1.
I20260619 04:45:24.045398 5469 raft_consensus.cc:2454] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ed9474500b54f3daae5c2c12c4914a0 in term 1.
I20260619 04:45:24.045600 5257 leader_election.cc:302] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed9474500b54f3daae5c2c12c4914a0, 99c906bb77a14d15838953a73feca757; no voters:
I20260619 04:45:24.045915 5679 raft_consensus.cc:2781] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:24.046139 5683 leader_election.cc:288] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [CANDIDATE]: Term 1 election: Requested vote from peers f17ce4cba4ae4763a6c6018ca5a0b406 (127.3.243.130:37689), 99c906bb77a14d15838953a73feca757 (127.3.243.131:38719)
I20260619 04:45:24.047874 5679 raft_consensus.cc:686] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 LEADER]: Becoming Leader. State: Replica: 1ed9474500b54f3daae5c2c12c4914a0, State: Running, Role: LEADER
I20260619 04:45:24.048223 5679 consensus_queue.cc:237] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.050884 5168 catalog_manager.cc:5495] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ed9474500b54f3daae5c2c12c4914a0 (127.3.243.129). New cstate: current_term: 1 leader_uuid: "1ed9474500b54f3daae5c2c12c4914a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } health_report { overall_health: UNKNOWN } } }
Jun 19 04:45:24 dist-test-slave-2qp8 krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844321, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:24.110204 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.094610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.3.243.190:35417)
0619 04:45:24.095574 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.095578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.095598 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:45:24.095641 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.095644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.095696 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.095786 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:24.097341 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.098149 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.100007 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.100437 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.100537 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.104745 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:24.104750 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:24.104753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:24.104781 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:24.107336 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.108194 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.108198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.108200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.108330 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.108725 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.108728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.108730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.108976 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:24.109081 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.109574 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.109954 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":199,"server-negotiator.queue_time_us":868,"thread_start_us":742,"threads_started":1}
I20260619 04:45:24.110328 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.094500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35417 (local address 127.0.0.1:55496)
0619 04:45:24.095200 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:24.095221 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:24.095386 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:24.095876 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:24.095882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:24.096811 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:24.097164 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.097177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.098276 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.098283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.099853 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.099876 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.100616 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.100623 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.100805 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.101505 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:24.101530 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:24.104552 (+ 3022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:24.107461 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:24.107472 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:24.107480 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:24.108023 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:24.108434 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:24.108439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:24.108446 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:24.108619 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:24.109096 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:24.109105 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:24.109311 (+ 206us) client_negotiation.cc:769] Sending connection context
0619 04:45:24.109501 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 04:45:24.109833 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":3,"thread_start_us":334,"threads_started":1}
I20260619 04:45:24.115535 5168 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:55496:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 04:45:24.116799 5168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:24.123775 5444 tablet_service.cc:1467] Processing CreateTablet for tablet a32c6e6c90954bfa8c0f743591916dc3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd58ed05349d4213b22453c794edfa9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:45:24.123803 5304 tablet_service.cc:1467] Processing CreateTablet for tablet a32c6e6c90954bfa8c0f743591916dc3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd58ed05349d4213b22453c794edfa9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:45:24.123926 5582 tablet_service.cc:1467] Processing CreateTablet for tablet a32c6e6c90954bfa8c0f743591916dc3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fd58ed05349d4213b22453c794edfa9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:45:24.124388 5304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32c6e6c90954bfa8c0f743591916dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:24.124405 5444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32c6e6c90954bfa8c0f743591916dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:24.124449 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32c6e6c90954bfa8c0f743591916dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:24.131539 5673 tablet_bootstrap.cc:492] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: Bootstrap starting.
I20260619 04:45:24.132328 5674 tablet_bootstrap.cc:492] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: Bootstrap starting.
I20260619 04:45:24.133337 5673 tablet_bootstrap.cc:654] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:24.134195 5674 tablet_bootstrap.cc:654] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:24.136462 5672 tablet_bootstrap.cc:492] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: Bootstrap starting.
I20260619 04:45:24.137957 5673 tablet_bootstrap.cc:492] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: No bootstrap required, opened a new log
I20260619 04:45:24.138181 5672 tablet_bootstrap.cc:654] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:24.138206 5673 ts_tablet_manager.cc:1397] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 04:45:24.138983 5673 raft_consensus.cc:348] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.139240 5673 raft_consensus.cc:374] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:24.139333 5673 raft_consensus.cc:729] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c906bb77a14d15838953a73feca757, State: Initialized, Role: FOLLOWER
I20260619 04:45:24.139648 5673 consensus_queue.cc:260] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.140254 5674 tablet_bootstrap.cc:492] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: No bootstrap required, opened a new log
I20260619 04:45:24.140139 5680 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.140424 5674 ts_tablet_manager.cc:1397] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260619 04:45:24.140470 5680 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:24.140911 5673 ts_tablet_manager.cc:1428] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:24.142207 5674 raft_consensus.cc:348] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.142527 5674 raft_consensus.cc:374] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:24.142633 5674 raft_consensus.cc:729] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ed9474500b54f3daae5c2c12c4914a0, State: Initialized, Role: FOLLOWER
I20260619 04:45:24.142897 5674 consensus_queue.cc:260] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.143383 5679 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.143697 5679 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:24.144809 5674 ts_tablet_manager.cc:1428] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260619 04:45:24.145076 5672 tablet_bootstrap.cc:492] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: No bootstrap required, opened a new log
I20260619 04:45:24.145249 5672 ts_tablet_manager.cc:1397] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260619 04:45:24.146039 5672 raft_consensus.cc:348] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.146766 5672 raft_consensus.cc:374] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:24.146843 5672 raft_consensus.cc:729] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f17ce4cba4ae4763a6c6018ca5a0b406, State: Initialized, Role: FOLLOWER
I20260619 04:45:24.147133 5672 consensus_queue.cc:260] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.147603 5678 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.147994 5678 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:24.148613 5672 ts_tablet_manager.cc:1428] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:24.266845 5678 raft_consensus.cc:482] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:24.267132 5678 raft_consensus.cc:504] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.268730 5678 leader_election.cc:288] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ed9474500b54f3daae5c2c12c4914a0 (127.3.243.129:40805), 99c906bb77a14d15838953a73feca757 (127.3.243.131:38719)
I20260619 04:45:24.277983 5713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.268994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:40805 (local address 127.3.243.130:47769)
0619 04:45:24.271001 (+ 2007us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:24.271020 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:24.271213 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:24.271769 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:24.271774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:24.271799 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:24.272220 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.272235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.273774 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.273779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.275205 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.275219 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.276990 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.276997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.277327 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.277379 (+ 52us) client_negotiation.cc:769] Sending connection context
0619 04:45:24.277486 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 04:45:24.277630 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1889,"thread_start_us":1205,"threads_started":1}
I20260619 04:45:24.278007 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.269262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:47769 (local address 127.3.243.129:40805)
0619 04:45:24.269512 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.269517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.271108 (+ 1591us) server_negotiation.cc:407] Connection header received
0619 04:45:24.271395 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.271400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.271477 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.271544 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:24.272405 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.273570 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.275361 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.276759 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.276939 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.277011 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.277540 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.277686 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:45:24.278033 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.269744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:51409 (local address 127.3.243.131:38719)
0619 04:45:24.269956 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.269963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.270820 (+ 857us) server_negotiation.cc:407] Connection header received
0619 04:45:24.271106 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.271111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.271181 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.271276 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:24.271845 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.272903 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.275558 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.276817 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.276948 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.277023 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.277639 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.277762 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:45:24.279083 5714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.268971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:38719 (local address 127.3.243.130:51409)
0619 04:45:24.270569 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:24.270631 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:24.270949 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:24.271312 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:24.271316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:24.271343 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:24.271686 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.271704 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.273037 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.273042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.275391 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.275406 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.277208 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.277217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.277433 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.277484 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 04:45:24.278752 (+ 1268us) client_negotiation.cc:240] Negotiation successful
0619 04:45:24.278857 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1427,"thread_start_us":1673,"threads_started":1}
I20260619 04:45:24.278935 5325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32c6e6c90954bfa8c0f743591916dc3" candidate_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed9474500b54f3daae5c2c12c4914a0" is_pre_election: true
I20260619 04:45:24.279459 5325 raft_consensus.cc:2454] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f17ce4cba4ae4763a6c6018ca5a0b406 in term 0.
I20260619 04:45:24.279785 5608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32c6e6c90954bfa8c0f743591916dc3" candidate_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c906bb77a14d15838953a73feca757" is_pre_election: true
I20260619 04:45:24.280064 5608 raft_consensus.cc:2454] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f17ce4cba4ae4763a6c6018ca5a0b406 in term 0.
I20260619 04:45:24.280164 5395 leader_election.cc:302] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ed9474500b54f3daae5c2c12c4914a0, f17ce4cba4ae4763a6c6018ca5a0b406; no voters:
I20260619 04:45:24.280603 5678 raft_consensus.cc:2781] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:24.280740 5678 raft_consensus.cc:482] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:24.280843 5678 raft_consensus.cc:3037] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:24.282919 5678 raft_consensus.cc:504] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.283542 5678 leader_election.cc:288] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [CANDIDATE]: Term 1 election: Requested vote from peers 1ed9474500b54f3daae5c2c12c4914a0 (127.3.243.129:40805), 99c906bb77a14d15838953a73feca757 (127.3.243.131:38719)
I20260619 04:45:24.283979 5608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32c6e6c90954bfa8c0f743591916dc3" candidate_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99c906bb77a14d15838953a73feca757"
I20260619 04:45:24.284003 5325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32c6e6c90954bfa8c0f743591916dc3" candidate_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ed9474500b54f3daae5c2c12c4914a0"
I20260619 04:45:24.284230 5608 raft_consensus.cc:3037] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:24.284269 5325 raft_consensus.cc:3037] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:24.286140 5608 raft_consensus.cc:2454] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f17ce4cba4ae4763a6c6018ca5a0b406 in term 1.
I20260619 04:45:24.286532 5397 leader_election.cc:302] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99c906bb77a14d15838953a73feca757, f17ce4cba4ae4763a6c6018ca5a0b406; no voters:
I20260619 04:45:24.286842 5678 raft_consensus.cc:2781] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:24.287386 5678 raft_consensus.cc:686] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 LEADER]: Becoming Leader. State: Replica: f17ce4cba4ae4763a6c6018ca5a0b406, State: Running, Role: LEADER
I20260619 04:45:24.287786 5678 consensus_queue.cc:237] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } }
I20260619 04:45:24.288187 5325 raft_consensus.cc:2454] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f17ce4cba4ae4763a6c6018ca5a0b406 in term 1.
I20260619 04:45:24.289857 5715 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: TxnStatusTablet state changed. Reason: New leader f17ce4cba4ae4763a6c6018ca5a0b406. Latest consensus state: current_term: 1 leader_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.290345 5715 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:45:24.291095 5717 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 04:45:24.291311 5717 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 04:45:24.291669 5167 catalog_manager.cc:5495] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 reported cstate change: term changed from 0 to 1, leader changed from <none> to f17ce4cba4ae4763a6c6018ca5a0b406 (127.3.243.130). New cstate: current_term: 1 leader_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } health_report { overall_health: UNKNOWN } } }
I20260619 04:45:24.309517 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.303406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:37689 (local address 127.0.0.1:35230)
0619 04:45:24.303736 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:24.303765 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:24.303877 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:24.304969 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:24.304973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:24.304985 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:45:24.305175 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.305183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.306206 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.306214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.307184 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.307195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.307881 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.307886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.308069 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.308200 (+ 131us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:45:24.308967 (+ 767us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:45:24.308997 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:45:24.309088 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 04:45:24.309244 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260619 04:45:24.309556 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.304101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.3.243.130:37689)
0619 04:45:24.304622 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.304626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.304642 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:45:24.304714 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.304718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.304786 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.304948 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:24.305355 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.306081 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.307330 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.307726 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.307857 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.308383 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:24.308860 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:24.309006 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.309126 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.309202 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":263,"threads_started":1}
I20260619 04:45:24.313966 5608 raft_consensus.cc:1261] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Refusing update from remote peer f17ce4cba4ae4763a6c6018ca5a0b406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:24.314198 5325 raft_consensus.cc:1261] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 FOLLOWER]: Refusing update from remote peer f17ce4cba4ae4763a6c6018ca5a0b406: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:24.314679 5678 consensus_queue.cc:1035] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:24.314960 5715 consensus_queue.cc:1035] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:24.327817 5679 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: TxnStatusTablet state changed. Reason: New leader f17ce4cba4ae4763a6c6018ca5a0b406. Latest consensus state: current_term: 1 leader_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.328172 5679 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:24.330242 5680 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: TxnStatusTablet state changed. Reason: New leader f17ce4cba4ae4763a6c6018ca5a0b406. Latest consensus state: current_term: 1 leader_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.330480 5680 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:24.333146 5715 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.333390 5715 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:45:24.333575 5688 mvcc.cc:205] Tried to move back new op lower bound from 7298434352380289024 to 7298434352286818304. Current Snapshot: MvccSnapshot[applied={T|T < 7298434352380289024 or (T in {7298434352380289024})}]
I20260619 04:45:24.333724 5687 mvcc.cc:205] Tried to move back new op lower bound from 7298434352380289024 to 7298434352286818304. Current Snapshot: MvccSnapshot[applied={T|T < 7298434352380289024}]
I20260619 04:45:24.335239 5680 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.335456 5680 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P 99c906bb77a14d15838953a73feca757: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:24.339327 5678 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.339622 5678 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P f17ce4cba4ae4763a6c6018ca5a0b406: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:45:24.341401 5679 tablet_replica.cc:439] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ed9474500b54f3daae5c2c12c4914a0" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40805 } } peers { permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 } } peers { permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 } } }
I20260619 04:45:24.341678 5679 tablet_replica.cc:442] T a32c6e6c90954bfa8c0f743591916dc3 P 1ed9474500b54f3daae5c2c12c4914a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:24.358058 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.352324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:46677 (local address 127.3.243.130:37689)
0619 04:45:24.353046 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.353053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.353126 (+ 73us) server_negotiation.cc:407] Connection header received
0619 04:45:24.353376 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.353385 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.353462 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.353564 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:24.354085 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.354965 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.356466 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.356893 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.357011 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.357410 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:24.357501 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:24.357596 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.357719 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.357786 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599}
I20260619 04:45:24.358238 5725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.352244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:37689 (local address 127.3.243.129:46677)
0619 04:45:24.353037 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:24.353058 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:24.353233 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:24.353581 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:24.353585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:24.353596 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:45:24.353920 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.353928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.355090 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.355099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.356330 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.356342 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.357078 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.357085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.357192 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.357239 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:45:24.357598 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:45:24.357609 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:45:24.357700 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 04:45:24.357827 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":407,"threads_started":1}
I20260619 04:45:24.369773 5608 raft_consensus.cc:1261] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Refusing update from remote peer 1ed9474500b54f3daae5c2c12c4914a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:24.370386 5469 raft_consensus.cc:1261] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Refusing update from remote peer 1ed9474500b54f3daae5c2c12c4914a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:24.370502 5679 consensus_queue.cc:1035] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99c906bb77a14d15838953a73feca757" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:24.370947 5701 consensus_queue.cc:1035] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f17ce4cba4ae4763a6c6018ca5a0b406" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:24.443607 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.424738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:46125 (local address 127.3.243.129:40805)
0619 04:45:24.425005 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.425012 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.434744 (+ 9732us) server_negotiation.cc:407] Connection header received
0619 04:45:24.435531 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.435537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.435639 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.435731 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:24.437917 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.439185 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.441306 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.441868 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.442052 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.442579 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:24.442721 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:24.442832 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.443172 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.443253 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260619 04:45:24.443842 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.424187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:40805 (local address 127.3.243.130:46125)
0619 04:45:24.434622 (+ 10435us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:24.434657 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:24.435313 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:24.436760 (+ 1447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:24.436767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:24.436787 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:45:24.437556 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.437576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.439496 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.439503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.441124 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:24.441137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.442073 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.442080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:24.442280 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.442369 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:45:24.442845 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:45:24.442859 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 04:45:24.443084 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 04:45:24.443240 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10151,"thread_start_us":10584,"threads_started":1}
I20260619 04:45:24.496381 5166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:24.496631 5166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:24.499553 5166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b4259d9ed7a0458fa38ae87749f9d892: Sending DeleteTablet for 3 replicas of tablet c99c2846220e47139106e2c1852ce72c
I20260619 04:45:24.500382 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet c99c2846220e47139106e2c1852ce72c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:24 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:35214
I20260619 04:45:24.500442 5304 tablet_service.cc:1514] Processing DeleteTablet for tablet c99c2846220e47139106e2c1852ce72c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:24 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:45762
I20260619 04:45:24.501286 5582 tablet_service.cc:1514] Processing DeleteTablet for tablet c99c2846220e47139106e2c1852ce72c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:24 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:42890
I20260619 04:45:24.515082 5733 tablet_replica.cc:330] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: stopping tablet replica
I20260619 04:45:24.515365 5733 raft_consensus.cc:2229] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:24.516412 5733 raft_consensus.cc:2258] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:24.517892 5733 ts_tablet_manager.cc:1905] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:24.524614 5736 tablet_replica.cc:330] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: stopping tablet replica
I20260619 04:45:24.524889 5736 raft_consensus.cc:2229] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:24.525219 5736 raft_consensus.cc:2258] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:24.526355 5736 ts_tablet_manager.cc:1905] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:24.526468 5733 ts_tablet_manager.cc:1918] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:45:24.526600 5733 log.cc:1196] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/c99c2846220e47139106e2c1852ce72c
I20260619 04:45:24.527127 5733 ts_tablet_manager.cc:1939] T c99c2846220e47139106e2c1852ce72c P f17ce4cba4ae4763a6c6018ca5a0b406: Deleting consensus metadata
I20260619 04:45:24.527593 5734 tablet_replica.cc:330] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: stopping tablet replica
I20260619 04:45:24.528316 5155 catalog_manager.cc:4841] TS f17ce4cba4ae4763a6c6018ca5a0b406 (127.3.243.130:37689): tablet c99c2846220e47139106e2c1852ce72c (table test-table [id=37a5d4a1f2214a1c954e0d8981c167ab]) successfully deleted
I20260619 04:45:24.528653 5734 raft_consensus.cc:2229] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:45:24.529006 5734 raft_consensus.cc:2258] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:24.532500 5734 ts_tablet_manager.cc:1905] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:24.539700 5736 ts_tablet_manager.cc:1918] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:45:24.539824 5736 log.cc:1196] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/c99c2846220e47139106e2c1852ce72c
I20260619 04:45:24.540287 5736 ts_tablet_manager.cc:1939] T c99c2846220e47139106e2c1852ce72c P 99c906bb77a14d15838953a73feca757: Deleting consensus metadata
I20260619 04:45:24.541303 5154 catalog_manager.cc:4841] TS 99c906bb77a14d15838953a73feca757 (127.3.243.131:38719): tablet c99c2846220e47139106e2c1852ce72c (table test-table [id=37a5d4a1f2214a1c954e0d8981c167ab]) successfully deleted
I20260619 04:45:24.546064 5734 ts_tablet_manager.cc:1918] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:45:24.546216 5734 log.cc:1196] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/c99c2846220e47139106e2c1852ce72c
I20260619 04:45:24.546639 5734 ts_tablet_manager.cc:1939] T c99c2846220e47139106e2c1852ce72c P 1ed9474500b54f3daae5c2c12c4914a0: Deleting consensus metadata
I20260619 04:45:24.549199 5152 catalog_manager.cc:4841] TS 1ed9474500b54f3daae5c2c12c4914a0 (127.3.243.129:40805): tablet c99c2846220e47139106e2c1852ce72c (table test-table [id=37a5d4a1f2214a1c954e0d8981c167ab]) successfully deleted
Jun 19 04:45:24 dist-test-slave-2qp8 krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:45:24.570205 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.542618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.3.243.129:40805)
0619 04:45:24.542811 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.542817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.549892 (+ 7075us) server_negotiation.cc:407] Connection header received
0619 04:45:24.550109 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.550114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.550189 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.550284 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:24.552185 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.553361 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.556647 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.557253 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.557338 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.562791 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:24.562807 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:24.562826 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:24.562866 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:24.566821 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.567689 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.567700 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.567704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.567876 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.568326 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.568333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.568338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.568798 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:24.569114 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.569394 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.569634 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260619 04:45:24.571012 5370 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45778
I20260619 04:45:24.588516 5709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.574329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.3.243.190:35417)
0619 04:45:24.574541 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.574547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.575008 (+ 461us) server_negotiation.cc:407] Connection header received
0619 04:45:24.575173 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.575179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.575263 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.575365 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:24.577019 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.578204 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.579373 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.580062 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.580224 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.582542 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:24.582551 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:24.582557 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:24.582597 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:24.585926 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.586601 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.586607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.586610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.586761 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.587076 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.587081 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.587084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.587430 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:24.587710 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.587904 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.588153 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 04:45:24.589133 5166 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55498
I20260619 04:45:24.591068 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5239
I20260619 04:45:24.605298 5670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.591948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.3.243.129:40805)
0619 04:45:24.592178 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.592186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.592643 (+ 457us) server_negotiation.cc:407] Connection header received
0619 04:45:24.592829 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.592835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.592917 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.593026 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:24.594226 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.595308 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.596497 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.597119 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.597255 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.599345 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:24.599351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:24.599357 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:24.599400 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:24.602719 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.603427 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.603433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.603437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.603591 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.603912 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.603917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.603920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.604312 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:24.604564 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.604748 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.604960 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 04:45:24.605933 5370 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45780
W20260619 04:45:24.606531 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5239: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:45:24.606693 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 5239
I20260619 04:45:24.635699 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5379
Jun 19 04:45:24 dist-test-slave-2qp8 krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:45:24.653844 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.636494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.3.243.130:37689)
0619 04:45:24.636729 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.636738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.636758 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:45:24.636930 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.636936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.637049 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.637184 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:24.638889 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.639674 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.641653 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.642266 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.642370 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.647547 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:24.647562 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:24.647581 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:24.647618 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:24.650819 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.651529 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.651536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.651539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.651680 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.652063 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.652069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.652073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.652563 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:24.652907 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.653203 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.653466 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260619 04:45:24.654592 5509 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35238
W20260619 04:45:24.655265 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5379: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:45:24.655445 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 5379
I20260619 04:45:24.674297 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5518
Jun 19 04:45:24 dist-test-slave-2qp8 krb5kdc[5123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:45:24.688700 5668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:24.675067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.3.243.131:38719)
0619 04:45:24.675329 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:24.675336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:24.675445 (+ 109us) server_negotiation.cc:407] Connection header received
0619 04:45:24.675682 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:24.675687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:24.675759 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:24.675885 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:24.677187 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.677979 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.679236 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:24.679688 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:24.679820 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:24.683779 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:24.683793 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:24.683809 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:24.683841 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:24.686459 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.687004 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.687019 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.687021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.687136 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:24.687435 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:24.687438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:24.687440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:24.687750 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:24.687975 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:24.688291 (+ 316us) server_negotiation.cc:299] Negotiation successful
0619 04:45:24.688450 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260619 04:45:24.689325 5648 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42896
W20260619 04:45:24.689901 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5518: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:45:24.690049 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 5518
I20260619 04:45:24.706193 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5136
I20260619 04:45:24.707293 5208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:24.822290 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 5136
2026-06-19T04:45:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4589 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:24 dist-test-slave-2qp8 krb5kdc[5744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:24 dist-test-slave-2qp8 krb5kdc[5744](info): set up 2 sockets
Jun 19 04:45:24 dist-test-slave-2qp8 krb5kdc[5744](info): commencing operation
krb5kdc: starting...
W20260619 04:45:25.896512 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:26 dist-test-slave-2qp8 krb5kdc[5744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:26.083806 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:39183
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:39183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:26.401574 5757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:26.401842 5757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:26.401928 5757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:26.409111 5757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:26.409214 5757 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 04:45:26.409303 5757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:26.409410 5757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:26.409521 5757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:26.409610 5757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:26.419181 5757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:39183
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:39183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:26.420496 5757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:26.421732 5757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:26.433777 5762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:26.433614 5765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:26.433614 5763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:26.434477 5757 server_base.cc:1034] running on GCE node
I20260619 04:45:26.435190 5757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:26.436630 5757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:26.437865 5757 hybrid_clock.cc:648] HybridClock initialized: now 1781844326437792 us; error 25 us; skew 500 ppm
I20260619 04:45:26.438151 5757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:26 dist-test-slave-2qp8 krb5kdc[5744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:26.441198 5757 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:26.442553 5757 webserver.cc:460] Webserver started at http://127.3.243.190:39105/ using document root <none> and password file <none>
I20260619 04:45:26.443198 5757 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:26.443316 5757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:26.443583 5757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:26.446589 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "9d51536b0f564b9787ca91433517e770"
format_stamp: "Formatted at 2026-06-19 04:45:26 on dist-test-slave-2qp8"
server_key: "fcfd4db5da2d9b6799ab57ffaf9c4ee2"
server_key_iv: "a3602b0a4ddd81dfc5fdae4285f852fc"
server_key_version: "encryptionkey@0"
I20260619 04:45:26.447293 5757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "9d51536b0f564b9787ca91433517e770"
format_stamp: "Formatted at 2026-06-19 04:45:26 on dist-test-slave-2qp8"
server_key: "fcfd4db5da2d9b6799ab57ffaf9c4ee2"
server_key_iv: "a3602b0a4ddd81dfc5fdae4285f852fc"
server_key_version: "encryptionkey@0"
I20260619 04:45:26.452541 5757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260619 04:45:26.455807 5772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:26.457125 5757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:26.457260 5757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "9d51536b0f564b9787ca91433517e770"
format_stamp: "Formatted at 2026-06-19 04:45:26 on dist-test-slave-2qp8"
server_key: "fcfd4db5da2d9b6799ab57ffaf9c4ee2"
server_key_iv: "a3602b0a4ddd81dfc5fdae4285f852fc"
server_key_version: "encryptionkey@0"
I20260619 04:45:26.457459 5757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:26.490211 5757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:26.494442 5757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:26.494668 5757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:26.515937 5757 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:39183
I20260619 04:45:26.515957 5833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:39183 every 8 connection(s)
I20260619 04:45:26.517360 5757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:26.520987 5834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:26.521621 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 5757
I20260619 04:45:26.521821 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:26.522447 4046 external_mini_cluster.cc:1428] Setting key d6d7679ff007b14db3817dd585b664c8
I20260619 04:45:26.534832 5834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770: Bootstrap starting.
Jun 19 04:45:26 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:26.538035 5834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:26.539395 5834 log.cc:824] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:26.542773 5834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770: No bootstrap required, opened a new log
I20260619 04:45:26.546967 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:26.524490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.3.243.190:39183)
0619 04:45:26.525326 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:26.525345 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:26.525397 (+ 52us) server_negotiation.cc:407] Connection header received
0619 04:45:26.527376 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:26.527404 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:26.527976 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:26.528492 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:26.529764 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:26.531709 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:26.533074 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:26.534066 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:26.534240 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:26.538490 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:26.538512 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:26.538532 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:26.538592 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:26.542501 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:26.543230 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:26.543239 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:26.543254 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:26.543460 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:26.543811 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:26.543816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:26.543819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:26.544580 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:26.544895 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:26.545209 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 04:45:26.545648 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":313,"threads_started":1}
I20260619 04:45:26.552382 5834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51536b0f564b9787ca91433517e770" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39183 } }
I20260619 04:45:26.552724 5834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:26.552796 5834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d51536b0f564b9787ca91433517e770, State: Initialized, Role: FOLLOWER
I20260619 04:45:26.553325 5834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51536b0f564b9787ca91433517e770" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39183 } }
I20260619 04:45:26.553596 5834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:26.553740 5834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:26.553926 5834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:26.556217 5834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51536b0f564b9787ca91433517e770" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39183 } }
I20260619 04:45:26.556736 5834 leader_election.cc:302] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d51536b0f564b9787ca91433517e770; no voters:
I20260619 04:45:26.557305 5834 leader_election.cc:288] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:26.557444 5839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:26.558143 5839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [term 1 LEADER]: Becoming Leader. State: Replica: 9d51536b0f564b9787ca91433517e770, State: Running, Role: LEADER
I20260619 04:45:26.558709 5839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51536b0f564b9787ca91433517e770" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39183 } }
I20260619 04:45:26.559326 5834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:26.561308 5840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d51536b0f564b9787ca91433517e770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51536b0f564b9787ca91433517e770" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39183 } } }
I20260619 04:45:26.561697 5840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:26.561462 5841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d51536b0f564b9787ca91433517e770. Latest consensus state: current_term: 1 leader_uuid: "9d51536b0f564b9787ca91433517e770" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d51536b0f564b9787ca91433517e770" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39183 } } }
I20260619 04:45:26.561950 5841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:26.563845 5844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:26.570019 5844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:26.587369 5844 catalog_manager.cc:1347] Generated new cluster ID: 93bddc1d8274436e801bb204ab44d441
I20260619 04:45:26.587514 5844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:26.608644 5844 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:26.609989 5844 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:26.624778 5844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770: Generated new TSK 0
I20260619 04:45:26.625676 5844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:45:26.672185 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:39183
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:27.005482 5860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:27.005782 5860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:27.005893 5860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:27.013135 5860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:27.013265 5860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:27.013357 5860 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 04:45:27.013411 5860 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:45:27.013484 5860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:27.023104 5860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.190:39183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:27.024453 5860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:27.025723 5860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:27.038451 5865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:27.038545 5866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:27.038599 5868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:27.039558 5860 server_base.cc:1034] running on GCE node
I20260619 04:45:27.040256 5860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:27.041265 5860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:27.042487 5860 hybrid_clock.cc:648] HybridClock initialized: now 1781844327042434 us; error 63 us; skew 500 ppm
I20260619 04:45:27.042806 5860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:27 dist-test-slave-2qp8 krb5kdc[5744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:27.045869 5860 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:45:27.047330 5860 webserver.cc:460] Webserver started at http://127.3.243.129:38151/ using document root <none> and password file <none>
I20260619 04:45:27.047989 5860 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:27.048192 5860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:27.048511 5860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:27.051465 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "11a1e9c0257c4138a28eb98fef8ffb22"
format_stamp: "Formatted at 2026-06-19 04:45:27 on dist-test-slave-2qp8"
server_key: "0326b169c1974276a499b3cad4a78f19"
server_key_iv: "e9014cf673307e8ef93ce5f223dd9a01"
server_key_version: "encryptionkey@0"
I20260619 04:45:27.052284 5860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "11a1e9c0257c4138a28eb98fef8ffb22"
format_stamp: "Formatted at 2026-06-19 04:45:27 on dist-test-slave-2qp8"
server_key: "0326b169c1974276a499b3cad4a78f19"
server_key_iv: "e9014cf673307e8ef93ce5f223dd9a01"
server_key_version: "encryptionkey@0"
I20260619 04:45:27.057288 5860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:45:27.060797 5875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:27.062104 5860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:45:27.062265 5860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "11a1e9c0257c4138a28eb98fef8ffb22"
format_stamp: "Formatted at 2026-06-19 04:45:27 on dist-test-slave-2qp8"
server_key: "0326b169c1974276a499b3cad4a78f19"
server_key_iv: "e9014cf673307e8ef93ce5f223dd9a01"
server_key_version: "encryptionkey@0"
I20260619 04:45:27.062462 5860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:27.082159 5860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:27.085996 5860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:27.086273 5860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:27.088972 5860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:27.089097 5860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:27.089267 5860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:27.089390 5860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:45:27 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:27.114530 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:27.089428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:46869 (local address 127.3.243.190:39183)
0619 04:45:27.089948 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:27.089956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:27.090990 (+ 1034us) server_negotiation.cc:407] Connection header received
0619 04:45:27.093006 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:27.093011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:27.093104 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:27.093222 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:27.095442 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.096524 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.099229 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.099894 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.100071 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:27.106024 (+ 5953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:27.106033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:27.106040 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:27.106091 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:27.110013 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:27.111199 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:27.111208 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:27.111212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:27.111435 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:27.112052 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:27.112059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:27.112063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:27.112435 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:27.112606 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:27.113663 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0619 04:45:27.113993 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":271,"threads_started":1}
I20260619 04:45:27.115321 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:27.090029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39183 (local address 127.3.243.129:46869)
0619 04:45:27.090835 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:27.090871 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:27.092739 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:27.093457 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:27.093464 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:27.094308 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:27.095248 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:27.095275 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.096670 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.096681 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:27.099015 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:27.099030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.100197 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.100207 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:27.100592 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:27.101375 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:27.101412 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:27.105698 (+ 4286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:27.110207 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:27.110224 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:27.110247 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:27.111038 (+ 791us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:27.111616 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:27.111623 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:27.111627 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:27.111914 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:27.112581 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:27.112590 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:27.113168 (+ 578us) client_negotiation.cc:769] Sending connection context
0619 04:45:27.113564 (+ 396us) client_negotiation.cc:240] Negotiation successful
0619 04:45:27.114083 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":364,"threads_started":1}
I20260619 04:45:27.136547 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:27.121514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39183 (local address 127.3.243.129:36315)
0619 04:45:27.122259 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:27.122281 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:27.122476 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:27.122845 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:27.122850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:27.123308 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:27.123629 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:27.123639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.124961 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.124967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:27.126034 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:27.126046 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.126852 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.126861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:27.127084 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:27.127884 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:27.127918 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:27.129317 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:27.133042 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:27.133047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:27.133052 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:27.133939 (+ 887us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:27.134381 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:27.134386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:27.134390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:27.134601 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:27.135567 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:27.135570 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:27.135736 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 04:45:27.135874 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 04:45:27.136195 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":353,"threads_started":1}
I20260619 04:45:27.136842 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:27.121590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:36315 (local address 127.3.243.190:39183)
0619 04:45:27.121873 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:27.121880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:27.122386 (+ 506us) server_negotiation.cc:407] Connection header received
0619 04:45:27.122632 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:27.122638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:27.122711 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:27.122821 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:27.123779 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.124783 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.126186 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.126614 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.126777 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:27.129547 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:27.129557 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:27.129563 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:27.129614 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:27.132894 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:27.134084 (+ 1190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:27.134090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:27.134095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:27.134252 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:27.134748 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:27.134757 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:27.134763 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:27.135378 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:27.135588 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:27.136031 (+ 443us) server_negotiation.cc:299] Negotiation successful
0619 04:45:27.136362 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 04:45:27.143997 5860 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:41929
I20260619 04:45:27.144122 5995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:41929 every 8 connection(s)
I20260619 04:45:27.145190 5860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:27.147104 5996 heartbeater.cc:347] Connected to a master server at 127.3.243.190:39183
I20260619 04:45:27.147349 5996 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:27.147935 5996 heartbeater.cc:511] Master 127.3.243.190:39183 requested a full tablet report, sending...
I20260619 04:45:27.149806 5789 ts_manager.cc:194] Registered new tserver with Master: 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929)
I20260619 04:45:27.151230 5789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:46869
I20260619 04:45:27.151769 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 5860
I20260619 04:45:27.151969 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:27.152745 4046 external_mini_cluster.cc:1428] Setting key 290c9b43ebbd685c8eb399e0fe8da533
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:45:27.191213 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:39183
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:27.505519 6000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:27.505786 6000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:27.505884 6000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:27.512804 6000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:27.512903 6000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:27.512986 6000 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 04:45:27.513037 6000 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:45:27.513098 6000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:27.522872 6000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.190:39183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:27.524205 6000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:27.525475 6000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:27.538204 6005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:27.538414 6006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:27.539188 6008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:27.539316 6000 server_base.cc:1034] running on GCE node
I20260619 04:45:27.539973 6000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:27.540902 6000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:27.542102 6000 hybrid_clock.cc:648] HybridClock initialized: now 1781844327542055 us; error 41 us; skew 500 ppm
I20260619 04:45:27.542421 6000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:27 dist-test-slave-2qp8 krb5kdc[5744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:27.545588 6000 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:45:27.546890 6000 webserver.cc:460] Webserver started at http://127.3.243.130:45945/ using document root <none> and password file <none>
I20260619 04:45:27.547509 6000 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:27.547605 6000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:27.547859 6000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:27.550817 6000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "6edb4355763f47a6a52593b0315097e9"
format_stamp: "Formatted at 2026-06-19 04:45:27 on dist-test-slave-2qp8"
server_key: "c382963132ea8098860502a2d7cef63b"
server_key_iv: "48b661294d24d2a3d815dce9918bd093"
server_key_version: "encryptionkey@0"
I20260619 04:45:27.551488 6000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "6edb4355763f47a6a52593b0315097e9"
format_stamp: "Formatted at 2026-06-19 04:45:27 on dist-test-slave-2qp8"
server_key: "c382963132ea8098860502a2d7cef63b"
server_key_iv: "48b661294d24d2a3d815dce9918bd093"
server_key_version: "encryptionkey@0"
I20260619 04:45:27.556335 6000 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 04:45:27.559402 6015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:27.560711 6000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:27.560914 6000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "6edb4355763f47a6a52593b0315097e9"
format_stamp: "Formatted at 2026-06-19 04:45:27 on dist-test-slave-2qp8"
server_key: "c382963132ea8098860502a2d7cef63b"
server_key_iv: "48b661294d24d2a3d815dce9918bd093"
server_key_version: "encryptionkey@0"
I20260619 04:45:27.561128 6000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:27.577636 6000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:27.581712 6000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:27.581977 6000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:27.584539 6000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:27.584681 6000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:27.584800 6000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:27.584892 6000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:45:27 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:27.607717 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:27.585166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:58087 (local address 127.3.243.190:39183)
0619 04:45:27.585356 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:27.585362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:27.586735 (+ 1373us) server_negotiation.cc:407] Connection header received
0619 04:45:27.588972 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:27.588977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:27.589068 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:27.589179 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:27.591254 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.592336 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.594720 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.595125 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.595255 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:27.600246 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:27.600253 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:27.600257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:27.600302 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:27.603785 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:27.604847 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:27.604856 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:27.604862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:27.605052 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:27.605537 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:27.605545 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:27.605551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:27.606033 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:27.606220 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:27.606871 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 04:45:27.607173 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 04:45:27.608348 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:27.585686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39183 (local address 127.3.243.130:58087)
0619 04:45:27.586549 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:27.586596 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:27.588603 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:27.589400 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:27.589412 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:27.590165 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:27.591067 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:27.591096 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.592501 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.592509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:27.594567 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:27.594582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.595378 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.595385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:27.595748 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:27.596534 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:27.596560 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:27.599887 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:27.603944 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:27.603953 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:27.603975 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:27.604706 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:27.605188 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:27.605198 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:27.605201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:27.605398 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:27.606162 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:27.606167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:27.606500 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 04:45:27.606711 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 04:45:27.607095 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":387,"threads_started":1}
I20260619 04:45:27.628219 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:27.614007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:46173 (local address 127.3.243.190:39183)
0619 04:45:27.614231 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:27.614240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:27.614858 (+ 618us) server_negotiation.cc:407] Connection header received
0619 04:45:27.615079 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:27.615087 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:27.615167 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:27.615284 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:27.616829 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.617806 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.619455 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.620252 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.620404 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:27.622177 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:27.622186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:27.622194 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:27.622241 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:27.625619 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:27.626275 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:27.626282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:27.626290 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:27.626404 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:27.626765 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:27.626771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:27.626774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:27.627079 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:27.627234 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:27.627617 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 04:45:27.627831 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:45:27.628335 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:27.613984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39183 (local address 127.3.243.130:46173)
0619 04:45:27.614730 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:27.614754 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:27.614926 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:27.615313 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:27.615318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:27.616184 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:27.616633 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:27.616650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.617969 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.617976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:27.619295 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:27.619312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:27.620466 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:27.620473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:27.620658 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:27.621281 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:27.621298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:27.621971 (+ 673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:27.625750 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:27.625757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:27.625761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:27.626152 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:27.626520 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:27.626525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:27.626529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:27.626660 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:27.627195 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:27.627200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:27.627378 (+ 178us) client_negotiation.cc:769] Sending connection context
0619 04:45:27.627520 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 04:45:27.627834 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":373,"threads_started":1}
I20260619 04:45:27.638741 6000 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:35043
I20260619 04:45:27.638814 6134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:35043 every 8 connection(s)
I20260619 04:45:27.640041 6000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:27.642117 6135 heartbeater.cc:347] Connected to a master server at 127.3.243.190:39183
I20260619 04:45:27.642341 6135 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:27.642963 6135 heartbeater.cc:511] Master 127.3.243.190:39183 requested a full tablet report, sending...
I20260619 04:45:27.644268 5789 ts_manager.cc:194] Registered new tserver with Master: 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043)
I20260619 04:45:27.645074 5789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:58087
I20260619 04:45:27.650027 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 6000
I20260619 04:45:27.650230 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:27.650635 4046 external_mini_cluster.cc:1428] Setting key e9a8bc1b18c0aab2ac2f2888fde4dc11
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:45:27.690826 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:39183
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:28.008091 6139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:28.008457 6139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:28.008668 6139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:28.016295 6139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:28.016471 6139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:28.016626 6139 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 04:45:28.016726 6139 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:45:28.016882 6139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:45:28.026926 6139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.190:39183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:28.028302 6139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:28.029744 6139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:28.043546 6147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:28.043520 6145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:28.043678 6144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:28.044279 6139 server_base.cc:1034] running on GCE node
I20260619 04:45:28.044890 6139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:28.045809 6139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:28.047039 6139 hybrid_clock.cc:648] HybridClock initialized: now 1781844328046988 us; error 43 us; skew 500 ppm
I20260619 04:45:28.047346 6139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:28 dist-test-slave-2qp8 krb5kdc[5744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844328, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:28.054263 6139 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:45:28.055703 6139 webserver.cc:460] Webserver started at http://127.3.243.131:46279/ using document root <none> and password file <none>
I20260619 04:45:28.056455 6139 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:28.056553 6139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:28.056805 6139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:28.059783 6139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "dd3869da6ad648fc8bbcccd310136089"
format_stamp: "Formatted at 2026-06-19 04:45:28 on dist-test-slave-2qp8"
server_key: "b4221b029a06a2613a2029cc09200aab"
server_key_iv: "9408eb85cbdc7b75efa802c10442ce66"
server_key_version: "encryptionkey@0"
I20260619 04:45:28.060627 6139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "dd3869da6ad648fc8bbcccd310136089"
format_stamp: "Formatted at 2026-06-19 04:45:28 on dist-test-slave-2qp8"
server_key: "b4221b029a06a2613a2029cc09200aab"
server_key_iv: "9408eb85cbdc7b75efa802c10442ce66"
server_key_version: "encryptionkey@0"
I20260619 04:45:28.066018 6139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:45:28.069595 6154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:28.070855 6139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 04:45:28.071029 6139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "dd3869da6ad648fc8bbcccd310136089"
format_stamp: "Formatted at 2026-06-19 04:45:28 on dist-test-slave-2qp8"
server_key: "b4221b029a06a2613a2029cc09200aab"
server_key_iv: "9408eb85cbdc7b75efa802c10442ce66"
server_key_version: "encryptionkey@0"
I20260619 04:45:28.071229 6139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:28.093997 6139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:28.098410 6139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:28.098706 6139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:28.101452 6139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:28.101608 6139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:28.101774 6139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:28.101898 6139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:45:28 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844328, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:28.125720 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.101879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:52527 (local address 127.3.243.190:39183)
0619 04:45:28.102081 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.102087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.103625 (+ 1538us) server_negotiation.cc:407] Connection header received
0619 04:45:28.105723 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.105732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.105840 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.105953 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:28.108223 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.109133 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.111781 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.112341 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.112474 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.117420 (+ 4946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:28.117428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:28.117433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:28.117519 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:28.121224 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:28.122303 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:28.122310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:28.122315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:28.122488 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:28.123138 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:28.123142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:28.123144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:28.123514 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:28.123674 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.124851 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.125171 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:45:28.126307 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.102566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39183 (local address 127.3.243.131:52527)
0619 04:45:28.103430 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.103477 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.105432 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.106214 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.106225 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.107093 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:28.108024 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.108054 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.109308 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.109314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.111548 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.111562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.112585 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.112592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.112885 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.113756 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:28.113804 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:28.117168 (+ 3364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:28.121397 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:28.121404 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:28.121417 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:28.122096 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:28.122630 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:28.122635 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:28.122639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:28.122975 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:28.123744 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:28.123749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:28.124320 (+ 571us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.124657 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.125113 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":380,"threads_started":1}
I20260619 04:45:28.151224 6273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:39149 every 8 connection(s)
I20260619 04:45:28.151235 6139 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:39149
I20260619 04:45:28.152077 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.132674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:34563 (local address 127.3.243.190:39183)
0619 04:45:28.132866 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.132873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.134450 (+ 1577us) server_negotiation.cc:407] Connection header received
0619 04:45:28.134669 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.134677 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.134767 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.134888 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:28.136683 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.137796 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.140479 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.140933 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.141122 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.143910 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:28.143921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:28.143930 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:28.143978 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:28.148301 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:28.149387 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:28.149395 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:28.149399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:28.149549 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:28.150047 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:28.150054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:28.150060 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:28.150509 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:28.150671 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.151444 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.151688 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:45:28.152927 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.133088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39183 (local address 127.3.243.131:34563)
0619 04:45:28.134323 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.134354 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.134520 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.134934 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.134945 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.135886 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:28.136482 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.136498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.137977 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.137986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.140314 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.140331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.141170 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.141177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.141400 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.142200 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:28.142223 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:28.143625 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:28.148527 (+ 4902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:28.148540 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:28.148552 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:28.149238 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:28.149683 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:28.149688 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:28.149692 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:28.149882 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:28.150798 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:28.150803 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:28.151066 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.152298 (+ 1232us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.152627 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"thread_start_us":862,"threads_started":1}
I20260619 04:45:28.153096 6139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:45:28.154629 5996 heartbeater.cc:503] Master 127.3.243.190:39183 was elected leader, sending a full tablet report...
I20260619 04:45:28.156579 6274 heartbeater.cc:347] Connected to a master server at 127.3.243.190:39183
I20260619 04:45:28.156785 6274 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:28.157480 6274 heartbeater.cc:511] Master 127.3.243.190:39183 requested a full tablet report, sending...
I20260619 04:45:28.158221 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 6139
I20260619 04:45:28.158414 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:45:28.158739 5789 ts_manager.cc:194] Registered new tserver with Master: dd3869da6ad648fc8bbcccd310136089 (127.3.243.131:39149)
I20260619 04:45:28.158977 4046 external_mini_cluster.cc:1428] Setting key 9e083128b02c884b100a03e6230a2081
I20260619 04:45:28.159729 5789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:52527
I20260619 04:45:28.162120 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 04:45:28 dist-test-slave-2qp8 krb5kdc[5744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:45:28 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:28.229329 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.207308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.3.243.190:39183)
0619 04:45:28.207489 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.207494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.207958 (+ 464us) server_negotiation.cc:407] Connection header received
0619 04:45:28.215369 (+ 7411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.215376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.215509 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.215632 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:28.217312 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.218199 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.219476 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.220201 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.220322 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.224496 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:28.224501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:28.224505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:28.224540 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:28.226870 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:28.227481 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:28.227495 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:28.227497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:28.227619 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:28.227960 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:28.227963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:28.227965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:28.228303 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:28.228499 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.228826 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.228997 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:45:28.233765 5789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:45:28.241029 5789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:28.274837 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.253210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.3.243.131:39149)
0619 04:45:28.263984 (+ 10774us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.263994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.265264 (+ 1270us) server_negotiation.cc:407] Connection header received
0619 04:45:28.265873 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.265880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.266133 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.266286 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:28.267780 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.269364 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.271760 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.273202 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.273330 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.273580 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.274291 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.274454 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10603,"thread_start_us":10772,"threads_started":1}
I20260619 04:45:28.274902 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.253178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41562 (local address 127.3.243.129:41929)
0619 04:45:28.264587 (+ 11409us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.264595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.264624 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:45:28.265631 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.265639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.265902 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.266073 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:28.267218 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.269134 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.272279 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.273754 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.273909 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.274114 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.274423 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.274588 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11250,"thread_start_us":11110,"threads_started":1}
I20260619 04:45:28.274893 6288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.252768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:39149 (local address 127.0.0.1:49026)
0619 04:45:28.265147 (+ 12379us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.265166 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.265352 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.266908 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.266914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.266937 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:28.267465 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.267479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.269530 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.269536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.271573 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.271586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.273715 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.273721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.273976 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.274065 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.274240 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.274454 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12251,"mutex_wait_us":98,"thread_start_us":12128,"threads_started":1}
I20260619 04:45:28.275746 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.256019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.3.243.130:35043)
0619 04:45:28.261656 (+ 5637us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.261668 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.267772 (+ 6104us) server_negotiation.cc:407] Connection header received
0619 04:45:28.267964 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.267973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.268204 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.268403 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:28.268989 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.270613 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.273141 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.274369 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.274501 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.274738 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.275319 (+ 581us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.275470 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5430,"thread_start_us":5259,"threads_started":1}
I20260619 04:45:28.276100 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.252614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41929 (local address 127.0.0.1:41562)
0619 04:45:28.264194 (+ 11580us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.264241 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.265279 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.266415 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.266423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.266455 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:28.266932 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.266944 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.269282 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.269290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.272099 (+ 2809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.272139 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.273971 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.273977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.274173 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.274206 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.274301 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.275089 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11269,"mutex_wait_us":688,"thread_start_us":11784,"threads_started":1}
I20260619 04:45:28.277263 6291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.255268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:35043 (local address 127.0.0.1:54856)
0619 04:45:28.267674 (+ 12406us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.267692 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.267796 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.268376 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.268380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.268403 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:28.268812 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.268823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.270789 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.270794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.272966 (+ 2172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.272978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.274918 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.274927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.275125 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.275154 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.275280 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.275398 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12283,"mutex_wait_us":2,"thread_start_us":12110,"threads_started":1}
I20260619 04:45:28.279165 5926 tablet_service.cc:1467] Processing CreateTablet for tablet 5be3542b62ab4f50a2c6c8b0a2118f00 (DEFAULT_TABLE table=test-table [id=e78754b148bb4143b79f30ba6b8834ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:28.279161 6204 tablet_service.cc:1467] Processing CreateTablet for tablet 5be3542b62ab4f50a2c6c8b0a2118f00 (DEFAULT_TABLE table=test-table [id=e78754b148bb4143b79f30ba6b8834ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:28.280309 6204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be3542b62ab4f50a2c6c8b0a2118f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:28.280318 5926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be3542b62ab4f50a2c6c8b0a2118f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:28.290302 6065 tablet_service.cc:1467] Processing CreateTablet for tablet 5be3542b62ab4f50a2c6c8b0a2118f00 (DEFAULT_TABLE table=test-table [id=e78754b148bb4143b79f30ba6b8834ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:28.291057 6293 tablet_bootstrap.cc:492] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: Bootstrap starting.
I20260619 04:45:28.291895 6065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5be3542b62ab4f50a2c6c8b0a2118f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:28.293880 6294 tablet_bootstrap.cc:492] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: Bootstrap starting.
I20260619 04:45:28.294395 6293 tablet_bootstrap.cc:654] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:28.296002 6293 log.cc:824] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:28.297297 6294 tablet_bootstrap.cc:654] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:28.298410 6294 log.cc:824] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:28.299010 6293 tablet_bootstrap.cc:492] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: No bootstrap required, opened a new log
I20260619 04:45:28.299259 6293 ts_tablet_manager.cc:1397] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 04:45:28.301321 6294 tablet_bootstrap.cc:492] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: No bootstrap required, opened a new log
I20260619 04:45:28.301565 6294 ts_tablet_manager.cc:1397] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 04:45:28.310103 6297 tablet_bootstrap.cc:492] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: Bootstrap starting.
I20260619 04:45:28.309810 6293 raft_consensus.cc:348] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } }
I20260619 04:45:28.310257 6293 raft_consensus.cc:374] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:28.310357 6293 raft_consensus.cc:729] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3869da6ad648fc8bbcccd310136089, State: Initialized, Role: FOLLOWER
I20260619 04:45:28.311089 6293 consensus_queue.cc:260] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } }
I20260619 04:45:28.311872 6294 raft_consensus.cc:348] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } }
I20260619 04:45:28.312335 6294 raft_consensus.cc:374] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:28.312443 6294 raft_consensus.cc:729] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a1e9c0257c4138a28eb98fef8ffb22, State: Initialized, Role: FOLLOWER
I20260619 04:45:28.312852 6274 heartbeater.cc:503] Master 127.3.243.190:39183 was elected leader, sending a full tablet report...
I20260619 04:45:28.313114 6294 consensus_queue.cc:260] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } }
I20260619 04:45:28.313330 6293 ts_tablet_manager.cc:1428] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260619 04:45:28.313627 6297 tablet_bootstrap.cc:654] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:28.314837 6297 log.cc:824] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:28.315056 6294 ts_tablet_manager.cc:1428] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 04:45:28.318372 6297 tablet_bootstrap.cc:492] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: No bootstrap required, opened a new log
I20260619 04:45:28.318646 6297 ts_tablet_manager.cc:1397] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:45:28.326005 6297 raft_consensus.cc:348] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } }
I20260619 04:45:28.326292 6297 raft_consensus.cc:374] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:28.326359 6297 raft_consensus.cc:729] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edb4355763f47a6a52593b0315097e9, State: Initialized, Role: FOLLOWER
I20260619 04:45:28.326798 6297 consensus_queue.cc:260] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } }
I20260619 04:45:28.327674 6135 heartbeater.cc:503] Master 127.3.243.190:39183 was elected leader, sending a full tablet report...
I20260619 04:45:28.328411 6297 ts_tablet_manager.cc:1428] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: Time spent starting tablet: real 0.010s user 0.006s sys 0.006s
W20260619 04:45:28.392388 6136 tablet.cc:2369] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:45:28.397526 5997 tablet.cc:2369] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:45:28.404727 6275 tablet.cc:2369] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:45:28.412534 6298 raft_consensus.cc:482] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:28.412809 6298 raft_consensus.cc:504] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } }
I20260619 04:45:28.414376 6298 leader_election.cc:288] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929), 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043)
I20260619 04:45:28.420481 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.414807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41929 (local address 127.3.243.131:35971)
0619 04:45:28.415149 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.415174 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.415307 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.415675 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.415679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.415704 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:28.415908 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.415916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.416991 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.416998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.418384 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.418435 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.419832 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.419838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.419979 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.420004 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.420079 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.420243 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":1,"spinlock_wait_cycles":29056}
I20260619 04:45:28.420529 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.414815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:35971 (local address 127.3.243.129:41929)
0619 04:45:28.415033 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.415039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.415309 (+ 270us) server_negotiation.cc:407] Connection header received
0619 04:45:28.415480 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.415487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.415552 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.415647 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:28.416062 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.416866 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.418581 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.419662 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.419824 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.419922 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.420149 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.420289 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:45:28.421339 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.415048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:60461 (local address 127.3.243.130:35043)
0619 04:45:28.415253 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.415259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.415689 (+ 430us) server_negotiation.cc:407] Connection header received
0619 04:45:28.415908 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.415912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.415985 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.416089 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:28.416599 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.417369 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.419244 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.420457 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.420630 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.420704 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.421037 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.421171 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:45:28.421478 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.414857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:35043 (local address 127.3.243.131:60461)
0619 04:45:28.415589 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.415603 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.415763 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.416100 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.416133 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.416157 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:28.416451 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.416460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.417512 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.417517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.419080 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.419092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.420686 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.420691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.420843 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.420877 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.420988 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.421143 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":420,"threads_started":1}
I20260619 04:45:28.421924 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be3542b62ab4f50a2c6c8b0a2118f00" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" is_pre_election: true
I20260619 04:45:28.422436 5946 raft_consensus.cc:2454] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 0.
I20260619 04:45:28.422407 6090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be3542b62ab4f50a2c6c8b0a2118f00" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edb4355763f47a6a52593b0315097e9" is_pre_election: true
I20260619 04:45:28.422937 6090 raft_consensus.cc:2454] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 0.
I20260619 04:45:28.423146 6157 leader_election.cc:302] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11a1e9c0257c4138a28eb98fef8ffb22, dd3869da6ad648fc8bbcccd310136089; no voters:
I20260619 04:45:28.423574 6298 raft_consensus.cc:2781] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:28.423717 6298 raft_consensus.cc:482] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:28.423834 6298 raft_consensus.cc:3037] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:28.426040 6298 raft_consensus.cc:504] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } }
I20260619 04:45:28.426666 6298 leader_election.cc:288] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 election: Requested vote from peers 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929), 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043)
I20260619 04:45:28.427157 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be3542b62ab4f50a2c6c8b0a2118f00" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a1e9c0257c4138a28eb98fef8ffb22"
I20260619 04:45:28.427182 6090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5be3542b62ab4f50a2c6c8b0a2118f00" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edb4355763f47a6a52593b0315097e9"
I20260619 04:45:28.427367 6090 raft_consensus.cc:3037] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:28.427367 5946 raft_consensus.cc:3037] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:28.430528 6090 raft_consensus.cc:2454] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 1.
I20260619 04:45:28.430539 5946 raft_consensus.cc:2454] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 1.
I20260619 04:45:28.430994 6156 leader_election.cc:302] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11a1e9c0257c4138a28eb98fef8ffb22, 6edb4355763f47a6a52593b0315097e9, dd3869da6ad648fc8bbcccd310136089; no voters:
I20260619 04:45:28.431339 6298 raft_consensus.cc:2781] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:28.431767 6298 raft_consensus.cc:686] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 1 LEADER]: Becoming Leader. State: Replica: dd3869da6ad648fc8bbcccd310136089, State: Running, Role: LEADER
I20260619 04:45:28.432317 6298 consensus_queue.cc:237] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } }
I20260619 04:45:28.436739 5789 catalog_manager.cc:5495] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd3869da6ad648fc8bbcccd310136089 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "dd3869da6ad648fc8bbcccd310136089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } health_report { overall_health: UNKNOWN } } }
Jun 19 04:45:28 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:28.491526 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.478403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.3.243.190:39183)
0619 04:45:28.478595 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.478599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.479109 (+ 510us) server_negotiation.cc:407] Connection header received
0619 04:45:28.479305 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.479313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.479376 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.479479 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:28.480712 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.481446 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.482705 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.483133 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.483273 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.486584 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:28.486588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:28.486591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:28.486617 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:28.488879 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:28.489551 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:28.489558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:28.489563 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:28.489692 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:28.490043 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:28.490050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:28.490055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:28.490333 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:28.490492 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.490886 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.491211 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":84}
I20260619 04:45:28.491549 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.478428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39183 (local address 127.0.0.1:40924)
0619 04:45:28.478996 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.479012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.479180 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.479499 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.479503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.480308 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:28.480560 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.480570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.481584 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.481592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.482560 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.482572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.483311 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.483318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.483476 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.484296 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:28.484313 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:28.486399 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:28.489017 (+ 2618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:28.489029 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:28.489036 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:28.489433 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:28.489808 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:28.489814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:28.489817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:28.489944 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:28.490457 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:28.490465 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:28.490640 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.490802 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.491085 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":320,"threads_started":1}
I20260619 04:45:28.496404 5789 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:40924:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 04:45:28.497285 5789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:28.503772 6065 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6d1a82e91f4e3bacef70d2bf06b31b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa521222acbf46e4bac92c615a2a4bea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:45:28.503638 5926 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6d1a82e91f4e3bacef70d2bf06b31b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa521222acbf46e4bac92c615a2a4bea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:45:28.503885 6204 tablet_service.cc:1467] Processing CreateTablet for tablet 8b6d1a82e91f4e3bacef70d2bf06b31b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fa521222acbf46e4bac92c615a2a4bea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:45:28.504669 5926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6d1a82e91f4e3bacef70d2bf06b31b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:28.504669 6065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6d1a82e91f4e3bacef70d2bf06b31b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:28.504828 6204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b6d1a82e91f4e3bacef70d2bf06b31b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:28.510402 6293 tablet_bootstrap.cc:492] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: Bootstrap starting.
I20260619 04:45:28.511776 6294 tablet_bootstrap.cc:492] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: Bootstrap starting.
I20260619 04:45:28.511940 6293 tablet_bootstrap.cc:654] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:28.512068 6297 tablet_bootstrap.cc:492] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: Bootstrap starting.
I20260619 04:45:28.513613 6297 tablet_bootstrap.cc:654] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:28.513684 6294 tablet_bootstrap.cc:654] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:28.515825 6297 tablet_bootstrap.cc:492] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: No bootstrap required, opened a new log
I20260619 04:45:28.515991 6297 ts_tablet_manager.cc:1397] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 04:45:28.516149 6294 tablet_bootstrap.cc:492] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: No bootstrap required, opened a new log
I20260619 04:45:28.516274 6294 ts_tablet_manager.cc:1397] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 04:45:28.516697 6297 raft_consensus.cc:348] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.516913 6297 raft_consensus.cc:374] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:28.517036 6297 raft_consensus.cc:729] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edb4355763f47a6a52593b0315097e9, State: Initialized, Role: FOLLOWER
I20260619 04:45:28.516932 6294 raft_consensus.cc:348] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.517143 6294 raft_consensus.cc:374] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:28.517223 6294 raft_consensus.cc:729] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a1e9c0257c4138a28eb98fef8ffb22, State: Initialized, Role: FOLLOWER
I20260619 04:45:28.517349 6297 consensus_queue.cc:260] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.517403 6294 consensus_queue.cc:260] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.517876 6294 ts_tablet_manager.cc:1428] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 04:45:28.517962 6297 ts_tablet_manager.cc:1428] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 04:45:28.517839 6301 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.518237 6301 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:28.518328 6299 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.518648 6299 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:28.518832 6293 tablet_bootstrap.cc:492] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: No bootstrap required, opened a new log
I20260619 04:45:28.518968 6293 ts_tablet_manager.cc:1397] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260619 04:45:28.519729 6293 raft_consensus.cc:348] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.519939 6293 raft_consensus.cc:374] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:28.520040 6293 raft_consensus.cc:729] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3869da6ad648fc8bbcccd310136089, State: Initialized, Role: FOLLOWER
I20260619 04:45:28.520278 6293 consensus_queue.cc:260] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.520674 6298 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.520946 6298 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:28.521920 6293 ts_tablet_manager.cc:1428] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:28.559707 6301 raft_consensus.cc:482] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:28.559937 6301 raft_consensus.cc:504] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.561697 6301 leader_election.cc:288] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929), dd3869da6ad648fc8bbcccd310136089 (127.3.243.131:39149)
I20260619 04:45:28.569046 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.562019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:45359 (local address 127.3.243.131:39149)
0619 04:45:28.562240 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.562245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.563044 (+ 799us) server_negotiation.cc:407] Connection header received
0619 04:45:28.563241 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.563246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.563324 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.563427 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:28.564176 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.564981 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.567156 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.567956 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.568190 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.568280 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.568637 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.568752 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:45:28.569217 6289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.561994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:56829 (local address 127.3.243.129:41929)
0619 04:45:28.562248 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.562252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.563243 (+ 991us) server_negotiation.cc:407] Connection header received
0619 04:45:28.563391 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.563398 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.563471 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.563573 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:28.564294 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.565150 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.567122 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.567949 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.568115 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.568223 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.568590 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.568715 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 04:45:28.569404 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.562377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41929 (local address 127.3.243.130:56829)
0619 04:45:28.563127 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.563140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.563276 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.563601 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.563604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.563644 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:28.564072 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.564095 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.565335 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.565339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.566972 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.566983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.568201 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.568207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.568395 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.568439 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.568528 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.568675 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":63,"spinlock_wait_cycles":1920,"thread_start_us":417,"threads_started":1}
I20260619 04:45:28.569599 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.561976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:39149 (local address 127.3.243.130:45359)
0619 04:45:28.562922 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.562946 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.563097 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.563445 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.563450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.563472 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:28.563931 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.563945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.565083 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.565088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.566981 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.566992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.568245 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.568251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.568457 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.568479 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.568595 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.568715 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":58,"spinlock_wait_cycles":2816,"thread_start_us":555,"threads_started":1}
I20260619 04:45:28.570058 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6d1a82e91f4e3bacef70d2bf06b31b" candidate_uuid: "6edb4355763f47a6a52593b0315097e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" is_pre_election: true
I20260619 04:45:28.570302 5946 raft_consensus.cc:2454] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edb4355763f47a6a52593b0315097e9 in term 0.
I20260619 04:45:28.570470 6228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6d1a82e91f4e3bacef70d2bf06b31b" candidate_uuid: "6edb4355763f47a6a52593b0315097e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3869da6ad648fc8bbcccd310136089" is_pre_election: true
I20260619 04:45:28.570788 6228 raft_consensus.cc:2454] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6edb4355763f47a6a52593b0315097e9 in term 0.
I20260619 04:45:28.571029 6018 leader_election.cc:302] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11a1e9c0257c4138a28eb98fef8ffb22, 6edb4355763f47a6a52593b0315097e9; no voters:
I20260619 04:45:28.571493 6301 raft_consensus.cc:2781] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:28.571617 6301 raft_consensus.cc:482] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:28.571729 6301 raft_consensus.cc:3037] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:28.573824 6301 raft_consensus.cc:504] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.574378 6301 leader_election.cc:288] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [CANDIDATE]: Term 1 election: Requested vote from peers 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929), dd3869da6ad648fc8bbcccd310136089 (127.3.243.131:39149)
I20260619 04:45:28.574841 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6d1a82e91f4e3bacef70d2bf06b31b" candidate_uuid: "6edb4355763f47a6a52593b0315097e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a1e9c0257c4138a28eb98fef8ffb22"
I20260619 04:45:28.574848 6228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b6d1a82e91f4e3bacef70d2bf06b31b" candidate_uuid: "6edb4355763f47a6a52593b0315097e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3869da6ad648fc8bbcccd310136089"
I20260619 04:45:28.575053 5946 raft_consensus.cc:3037] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:28.575040 6228 raft_consensus.cc:3037] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:28.577147 5946 raft_consensus.cc:2454] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edb4355763f47a6a52593b0315097e9 in term 1.
I20260619 04:45:28.577315 6228 raft_consensus.cc:2454] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6edb4355763f47a6a52593b0315097e9 in term 1.
I20260619 04:45:28.577546 6018 leader_election.cc:302] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11a1e9c0257c4138a28eb98fef8ffb22, 6edb4355763f47a6a52593b0315097e9; no voters:
I20260619 04:45:28.577811 6301 raft_consensus.cc:2781] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:28.578207 6301 raft_consensus.cc:686] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6edb4355763f47a6a52593b0315097e9, State: Running, Role: LEADER
I20260619 04:45:28.578660 6301 consensus_queue.cc:237] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.581931 6316 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: TxnStatusTablet state changed. Reason: New leader 6edb4355763f47a6a52593b0315097e9. Latest consensus state: current_term: 1 leader_uuid: "6edb4355763f47a6a52593b0315097e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.582309 6316 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:45:28.582993 6318 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 04:45:28.583264 6318 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 04:45:28.583287 5789 catalog_manager.cc:5495] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6edb4355763f47a6a52593b0315097e9 (127.3.243.130). New cstate: current_term: 1 leader_uuid: "6edb4355763f47a6a52593b0315097e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } health_report { overall_health: UNKNOWN } } }
I20260619 04:45:28.619603 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.613784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54868 (local address 127.3.243.130:35043)
0619 04:45:28.613951 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.613957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.614156 (+ 199us) server_negotiation.cc:407] Connection header received
0619 04:45:28.614444 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.614448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.614541 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.614695 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:28.615061 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.615848 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.617340 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.617748 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.617888 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.618399 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:28.618934 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:28.619083 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.619167 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.619245 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:45:28.619585 6310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.613820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:35043 (local address 127.0.0.1:54868)
0619 04:45:28.614066 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.614082 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.614244 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.614714 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.614719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.614731 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:45:28.614928 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.614938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.615983 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.615992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.617206 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.617224 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.617903 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.617909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.618130 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.618204 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:45:28.619048 (+ 844us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:45:28.619068 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.619148 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.619291 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 04:45:28.627422 6228 raft_consensus.cc:1261] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Refusing update from remote peer 6edb4355763f47a6a52593b0315097e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:28.627635 5946 raft_consensus.cc:1261] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Refusing update from remote peer 6edb4355763f47a6a52593b0315097e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:28.628357 6316 consensus_queue.cc:1035] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:28.628738 6301 consensus_queue.cc:1035] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:28.635691 6298 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: TxnStatusTablet state changed. Reason: New leader 6edb4355763f47a6a52593b0315097e9. Latest consensus state: current_term: 1 leader_uuid: "6edb4355763f47a6a52593b0315097e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.636076 6298 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:28.640439 6299 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: TxnStatusTablet state changed. Reason: New leader 6edb4355763f47a6a52593b0315097e9. Latest consensus state: current_term: 1 leader_uuid: "6edb4355763f47a6a52593b0315097e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.640700 6299 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:28.648339 6321 mvcc.cc:205] Tried to move back new op lower bound from 7298434370049769472 to 7298434369862189056. Current Snapshot: MvccSnapshot[applied={T|T < 7298434370049769472}]
I20260619 04:45:28.654445 6298 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6edb4355763f47a6a52593b0315097e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.654703 6298 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P dd3869da6ad648fc8bbcccd310136089: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:28.657287 6301 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6edb4355763f47a6a52593b0315097e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.657560 6301 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:45:28.658445 6316 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6edb4355763f47a6a52593b0315097e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.658659 6316 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 6edb4355763f47a6a52593b0315097e9: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:45:28.676214 6299 tablet_replica.cc:439] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6edb4355763f47a6a52593b0315097e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } } }
I20260619 04:45:28.676496 6299 tablet_replica.cc:442] T 8b6d1a82e91f4e3bacef70d2bf06b31b P 11a1e9c0257c4138a28eb98fef8ffb22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:45:28.682883 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.675034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.3.243.131:39149)
0619 04:45:28.675216 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.675232 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.677883 (+ 2651us) server_negotiation.cc:407] Connection header received
0619 04:45:28.678145 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.678150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.678232 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.678375 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:28.678737 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.679459 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.680863 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.681241 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.681388 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.681799 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:28.682278 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:28.682474 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.682533 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.682611 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 04:45:28.699142 6333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.693152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:35043 (local address 127.3.243.131:33881)
0619 04:45:28.693818 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.693834 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.693946 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.694290 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.694294 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.694302 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:45:28.694554 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.694561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.695966 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.695969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.697063 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.697074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.698091 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.698095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.698200 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.698247 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:45:28.698685 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:45:28.698693 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.698751 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.698845 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":355,"threads_started":1}
I20260619 04:45:28.699172 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.693200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:33881 (local address 127.3.243.130:35043)
0619 04:45:28.693425 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.693431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.693915 (+ 484us) server_negotiation.cc:407] Connection header received
0619 04:45:28.694073 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.694078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.694171 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.694271 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:28.694689 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.695849 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.697201 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.697920 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.698057 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.698450 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:28.698586 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:28.698689 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.698799 (+ 110us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.698878 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:45:28.708719 6090 raft_consensus.cc:1261] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Refusing update from remote peer dd3869da6ad648fc8bbcccd310136089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:28.709354 5946 raft_consensus.cc:1261] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Refusing update from remote peer dd3869da6ad648fc8bbcccd310136089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:28.709498 6303 consensus_queue.cc:1035] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:28.709900 6298 consensus_queue.cc:1035] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:28.715354 6322 mvcc.cc:205] Tried to move back new op lower bound from 7298434370383777792 to 7298434369262485504. Current Snapshot: MvccSnapshot[applied={T|T < 7298434370383777792}]
I20260619 04:45:28.747715 6290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.740573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:45051 (local address 127.3.243.131:39149)
0619 04:45:28.740811 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:28.740820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:28.741275 (+ 455us) server_negotiation.cc:407] Connection header received
0619 04:45:28.741448 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:28.741452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:28.741519 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:28.741589 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:28.742262 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.743384 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.745665 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.746306 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.746449 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.746945 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:28.747073 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:28.747138 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:28.747355 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 04:45:28.747438 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 04:45:28.747753 6337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:28.740515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:39149 (local address 127.3.243.130:45051)
0619 04:45:28.741164 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:28.741189 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:28.741331 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:28.741626 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:28.741632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:28.741644 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:45:28.742058 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.742070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.743579 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.743586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.745435 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:28.745450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:28.746496 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:28.746513 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:28.746667 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:28.746722 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:45:28.747221 (+ 499us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:45:28.747231 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 04:45:28.747312 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 04:45:28.747439 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":312,"threads_started":1}
I20260619 04:45:28.786247 5787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:28.786541 5787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:28.790436 5787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770: Sending DeleteTablet for 3 replicas of tablet 5be3542b62ab4f50a2c6c8b0a2118f00
I20260619 04:45:28.791543 5926 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be3542b62ab4f50a2c6c8b0a2118f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:28 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:41562
I20260619 04:45:28.791968 6204 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be3542b62ab4f50a2c6c8b0a2118f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:28 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:49026
I20260619 04:45:28.792649 6065 tablet_service.cc:1514] Processing DeleteTablet for tablet 5be3542b62ab4f50a2c6c8b0a2118f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:28 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:54856
I20260619 04:45:28.797384 5787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:45:28.798050 5787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:28.804281 6341 tablet_replica.cc:330] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: stopping tablet replica
I20260619 04:45:28.804811 6341 raft_consensus.cc:2229] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:28.805284 6341 raft_consensus.cc:2258] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:28.807425 6341 ts_tablet_manager.cc:1905] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:28.811458 5926 tablet_service.cc:1467] Processing CreateTablet for tablet 227ef2966b9842d19a747e44882af879 (DEFAULT_TABLE table=test-table [id=464d7f90114e4ee1aa3c1bb9fb927745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:28.811488 6203 tablet_service.cc:1467] Processing CreateTablet for tablet 227ef2966b9842d19a747e44882af879 (DEFAULT_TABLE table=test-table [id=464d7f90114e4ee1aa3c1bb9fb927745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:28.811920 5926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227ef2966b9842d19a747e44882af879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:28.811972 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227ef2966b9842d19a747e44882af879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:28.815867 6341 ts_tablet_manager.cc:1918] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:45:28.816133 6341 log.cc:1196] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/5be3542b62ab4f50a2c6c8b0a2118f00
I20260619 04:45:28.816766 6341 ts_tablet_manager.cc:1939] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting consensus metadata
I20260619 04:45:28.818724 6293 tablet_bootstrap.cc:492] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: Bootstrap starting.
I20260619 04:45:28.818710 5775 catalog_manager.cc:4841] TS 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929): tablet 5be3542b62ab4f50a2c6c8b0a2118f00 (table test-table [id=e78754b148bb4143b79f30ba6b8834ab]) successfully deleted
I20260619 04:45:28.820909 6293 tablet_bootstrap.cc:654] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:28.823572 6064 tablet_service.cc:1467] Processing CreateTablet for tablet 227ef2966b9842d19a747e44882af879 (DEFAULT_TABLE table=test-table [id=464d7f90114e4ee1aa3c1bb9fb927745]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:28.824167 6064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227ef2966b9842d19a747e44882af879. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:28.824421 6293 tablet_bootstrap.cc:492] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: No bootstrap required, opened a new log
I20260619 04:45:28.824635 6293 ts_tablet_manager.cc:1397] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 04:45:28.825173 6294 tablet_bootstrap.cc:492] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: Bootstrap starting.
I20260619 04:45:28.825613 6343 tablet_replica.cc:330] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: stopping tablet replica
I20260619 04:45:28.826116 6343 raft_consensus.cc:2229] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:28.826498 6343 raft_consensus.cc:2258] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:28.826737 6294 tablet_bootstrap.cc:654] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:28.827234 6342 tablet_replica.cc:330] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: stopping tablet replica
I20260619 04:45:28.827653 6342 raft_consensus.cc:2229] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:45:28.829496 6342 raft_consensus.cc:2258] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:28.829926 6293 raft_consensus.cc:348] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.830273 6293 raft_consensus.cc:374] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:28.830394 6293 raft_consensus.cc:729] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3869da6ad648fc8bbcccd310136089, State: Initialized, Role: FOLLOWER
I20260619 04:45:28.830480 6343 ts_tablet_manager.cc:1905] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:28.832098 6297 tablet_bootstrap.cc:492] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: Bootstrap starting.
I20260619 04:45:28.832422 6293 consensus_queue.cc:260] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.834026 6297 tablet_bootstrap.cc:654] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:28.834167 6293 ts_tablet_manager.cc:1428] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260619 04:45:28.839078 6342 ts_tablet_manager.cc:1905] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:28.843644 6294 tablet_bootstrap.cc:492] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: No bootstrap required, opened a new log
I20260619 04:45:28.845084 6294 ts_tablet_manager.cc:1397] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.004s
I20260619 04:45:28.846140 6294 raft_consensus.cc:348] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.846482 6294 raft_consensus.cc:374] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:28.846589 6294 raft_consensus.cc:729] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a1e9c0257c4138a28eb98fef8ffb22, State: Initialized, Role: FOLLOWER
I20260619 04:45:28.846858 6342 ts_tablet_manager.cc:1918] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:45:28.847047 6342 log.cc:1196] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/5be3542b62ab4f50a2c6c8b0a2118f00
I20260619 04:45:28.846974 6294 consensus_queue.cc:260] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.847663 6342 ts_tablet_manager.cc:1939] T 5be3542b62ab4f50a2c6c8b0a2118f00 P dd3869da6ad648fc8bbcccd310136089: Deleting consensus metadata
I20260619 04:45:28.849337 6343 ts_tablet_manager.cc:1918] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:45:28.849537 6343 log.cc:1196] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/5be3542b62ab4f50a2c6c8b0a2118f00
I20260619 04:45:28.849809 6297 tablet_bootstrap.cc:492] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: No bootstrap required, opened a new log
I20260619 04:45:28.850000 6297 ts_tablet_manager.cc:1397] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20260619 04:45:28.850122 6343 ts_tablet_manager.cc:1939] T 5be3542b62ab4f50a2c6c8b0a2118f00 P 6edb4355763f47a6a52593b0315097e9: Deleting consensus metadata
I20260619 04:45:28.850333 6294 ts_tablet_manager.cc:1428] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260619 04:45:28.850786 6297 raft_consensus.cc:348] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.851092 6297 raft_consensus.cc:374] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:28.851199 6297 raft_consensus.cc:729] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edb4355763f47a6a52593b0315097e9, State: Initialized, Role: FOLLOWER
I20260619 04:45:28.851490 6297 consensus_queue.cc:260] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.851939 5774 catalog_manager.cc:4841] TS 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043): tablet 5be3542b62ab4f50a2c6c8b0a2118f00 (table test-table [id=e78754b148bb4143b79f30ba6b8834ab]) successfully deleted
I20260619 04:45:28.853282 6297 ts_tablet_manager.cc:1428] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:28.853857 5773 catalog_manager.cc:4841] TS dd3869da6ad648fc8bbcccd310136089 (127.3.243.131:39149): tablet 5be3542b62ab4f50a2c6c8b0a2118f00 (table test-table [id=e78754b148bb4143b79f30ba6b8834ab]) successfully deleted
I20260619 04:45:28.898041 6303 raft_consensus.cc:482] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:28.898367 6303 raft_consensus.cc:504] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.899212 6303 leader_election.cc:288] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929), 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043)
I20260619 04:45:28.899921 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227ef2966b9842d19a747e44882af879" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" is_pre_election: true
I20260619 04:45:28.900074 6090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227ef2966b9842d19a747e44882af879" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edb4355763f47a6a52593b0315097e9" is_pre_election: true
I20260619 04:45:28.900204 5946 raft_consensus.cc:2454] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 0.
I20260619 04:45:28.900398 6090 raft_consensus.cc:2454] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 0.
I20260619 04:45:28.900945 6156 leader_election.cc:302] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edb4355763f47a6a52593b0315097e9, dd3869da6ad648fc8bbcccd310136089; no voters:
I20260619 04:45:28.901434 6303 raft_consensus.cc:2781] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:28.901546 6303 raft_consensus.cc:482] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:28.901660 6303 raft_consensus.cc:3037] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:28.905015 6303 raft_consensus.cc:504] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.906164 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227ef2966b9842d19a747e44882af879" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a1e9c0257c4138a28eb98fef8ffb22"
I20260619 04:45:28.906363 5946 raft_consensus.cc:3037] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:28.906365 6090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227ef2966b9842d19a747e44882af879" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edb4355763f47a6a52593b0315097e9"
I20260619 04:45:28.906579 6090 raft_consensus.cc:3037] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:28.909371 5946 raft_consensus.cc:2454] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 1.
I20260619 04:45:28.909868 6157 leader_election.cc:302] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11a1e9c0257c4138a28eb98fef8ffb22, dd3869da6ad648fc8bbcccd310136089; no voters:
I20260619 04:45:28.909889 6090 raft_consensus.cc:2454] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 1.
I20260619 04:45:28.910187 6298 raft_consensus.cc:2781] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:28.910329 6298 raft_consensus.cc:686] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 1 LEADER]: Becoming Leader. State: Replica: dd3869da6ad648fc8bbcccd310136089, State: Running, Role: LEADER
I20260619 04:45:28.910624 6298 consensus_queue.cc:237] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:28.911276 6303 leader_election.cc:288] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 election: Requested vote from peers 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929), 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043)
I20260619 04:45:28.913777 5789 catalog_manager.cc:5495] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd3869da6ad648fc8bbcccd310136089 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "dd3869da6ad648fc8bbcccd310136089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } health_report { overall_health: HEALTHY } } }
I20260619 04:45:28.948443 5946 raft_consensus.cc:1261] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Refusing update from remote peer dd3869da6ad648fc8bbcccd310136089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:28.948529 6090 raft_consensus.cc:1261] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Refusing update from remote peer dd3869da6ad648fc8bbcccd310136089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:28.949054 6298 consensus_queue.cc:1035] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:28.949306 6303 consensus_queue.cc:1035] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:28.967140 6323 mvcc.cc:205] Tried to move back new op lower bound from 7298434371367153664 to 7298434371220500480. Current Snapshot: MvccSnapshot[applied={T|T < 7298434371367153664}]
W20260619 04:45:28.987432 6322 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 04:45:30.583009 5789 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:30.583256 5789 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:30.586933 5789 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770: Sending DeleteTablet for 3 replicas of tablet 227ef2966b9842d19a747e44882af879
I20260619 04:45:30.587656 5926 tablet_service.cc:1514] Processing DeleteTablet for tablet 227ef2966b9842d19a747e44882af879 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:30 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:41562
I20260619 04:45:30.587778 6064 tablet_service.cc:1514] Processing DeleteTablet for tablet 227ef2966b9842d19a747e44882af879 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:30 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:54856
I20260619 04:45:30.587873 6204 tablet_service.cc:1514] Processing DeleteTablet for tablet 227ef2966b9842d19a747e44882af879 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:30 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:49026
I20260619 04:45:30.598711 6373 tablet_replica.cc:330] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: stopping tablet replica
I20260619 04:45:30.599107 6373 raft_consensus.cc:2229] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:45:30.599593 6373 raft_consensus.cc:2258] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:30.600088 6371 tablet_replica.cc:330] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: stopping tablet replica
I20260619 04:45:30.600488 6371 raft_consensus.cc:2229] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:30.600744 6372 tablet_replica.cc:330] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: stopping tablet replica
I20260619 04:45:30.600802 6371 raft_consensus.cc:2258] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:30.600816 6373 ts_tablet_manager.cc:1905] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:30.601013 6372 raft_consensus.cc:2229] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:30.601306 6372 raft_consensus.cc:2258] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:30.601817 6371 ts_tablet_manager.cc:1905] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:30.602262 6372 ts_tablet_manager.cc:1905] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:30.606724 6371 ts_tablet_manager.cc:1918] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:45:30.606890 6371 log.cc:1196] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/227ef2966b9842d19a747e44882af879
I20260619 04:45:30.607367 6371 ts_tablet_manager.cc:1939] T 227ef2966b9842d19a747e44882af879 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting consensus metadata
I20260619 04:45:30.607724 6373 ts_tablet_manager.cc:1918] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:45:30.607896 6373 log.cc:1196] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/227ef2966b9842d19a747e44882af879
I20260619 04:45:30.608345 6373 ts_tablet_manager.cc:1939] T 227ef2966b9842d19a747e44882af879 P dd3869da6ad648fc8bbcccd310136089: Deleting consensus metadata
I20260619 04:45:30.608609 5775 catalog_manager.cc:4841] TS 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929): tablet 227ef2966b9842d19a747e44882af879 (table test-table [id=464d7f90114e4ee1aa3c1bb9fb927745]) successfully deleted
I20260619 04:45:30.609117 6372 ts_tablet_manager.cc:1918] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:45:30.609401 6372 log.cc:1196] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/227ef2966b9842d19a747e44882af879
I20260619 04:45:30.609423 5773 catalog_manager.cc:4841] TS dd3869da6ad648fc8bbcccd310136089 (127.3.243.131:39149): tablet 227ef2966b9842d19a747e44882af879 (table test-table [id=464d7f90114e4ee1aa3c1bb9fb927745]) successfully deleted
I20260619 04:45:30.609817 6372 ts_tablet_manager.cc:1939] T 227ef2966b9842d19a747e44882af879 P 6edb4355763f47a6a52593b0315097e9: Deleting consensus metadata
I20260619 04:45:30.610781 5774 catalog_manager.cc:4841] TS 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043): tablet 227ef2966b9842d19a747e44882af879 (table test-table [id=464d7f90114e4ee1aa3c1bb9fb927745]) successfully deleted
Jun 19 04:45:30 dist-test-slave-2qp8 krb5kdc[5744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:45:30 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:30.647758 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:30.633436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40940 (local address 127.3.243.190:39183)
0619 04:45:30.633972 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:30.633976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:30.633997 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:45:30.634165 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:30.634168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:30.634230 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:30.634318 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:30.635542 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:30.636379 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:30.637568 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:30.637948 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:30.638041 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:30.642417 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:30.642421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:30.642424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:30.642456 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:30.645129 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:30.645774 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:30.645787 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:30.645790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:30.645913 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:30.646298 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:30.646302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:30.646305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:30.646573 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:30.646844 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:30.647170 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 04:45:30.647383 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":290,"threads_started":1}
I20260619 04:45:30.651168 5789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:45:30.651774 5789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:30.658335 5926 tablet_service.cc:1467] Processing CreateTablet for tablet 34f58e65f77342c896e35dd3124166b7 (DEFAULT_TABLE table=test-table [id=95b691794e9042a99a3ead9df35b4929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:30.658521 6064 tablet_service.cc:1467] Processing CreateTablet for tablet 34f58e65f77342c896e35dd3124166b7 (DEFAULT_TABLE table=test-table [id=95b691794e9042a99a3ead9df35b4929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:30.658869 5926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f58e65f77342c896e35dd3124166b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:30.658943 6064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f58e65f77342c896e35dd3124166b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:30.661789 6204 tablet_service.cc:1467] Processing CreateTablet for tablet 34f58e65f77342c896e35dd3124166b7 (DEFAULT_TABLE table=test-table [id=95b691794e9042a99a3ead9df35b4929]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:30.662221 6204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f58e65f77342c896e35dd3124166b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:30.670483 6382 tablet_bootstrap.cc:492] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: Bootstrap starting.
I20260619 04:45:30.672079 6382 tablet_bootstrap.cc:654] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:30.672123 6384 tablet_bootstrap.cc:492] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: Bootstrap starting.
I20260619 04:45:30.672778 6383 tablet_bootstrap.cc:492] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: Bootstrap starting.
I20260619 04:45:30.673710 6384 tablet_bootstrap.cc:654] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:30.674439 6382 tablet_bootstrap.cc:492] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: No bootstrap required, opened a new log
I20260619 04:45:30.674557 6383 tablet_bootstrap.cc:654] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:30.674580 6382 ts_tablet_manager.cc:1397] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 04:45:30.675442 6382 raft_consensus.cc:348] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:30.675793 6382 raft_consensus.cc:374] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:30.675951 6382 raft_consensus.cc:729] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edb4355763f47a6a52593b0315097e9, State: Initialized, Role: FOLLOWER
I20260619 04:45:30.676277 6382 consensus_queue.cc:260] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:30.676505 6384 tablet_bootstrap.cc:492] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: No bootstrap required, opened a new log
I20260619 04:45:30.676676 6384 ts_tablet_manager.cc:1397] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 04:45:30.676826 6382 ts_tablet_manager.cc:1428] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 04:45:30.676950 6383 tablet_bootstrap.cc:492] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: No bootstrap required, opened a new log
I20260619 04:45:30.677109 6383 ts_tablet_manager.cc:1397] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260619 04:45:30.677518 6384 raft_consensus.cc:348] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:30.677747 6384 raft_consensus.cc:374] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:30.677863 6384 raft_consensus.cc:729] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3869da6ad648fc8bbcccd310136089, State: Initialized, Role: FOLLOWER
I20260619 04:45:30.677943 6383 raft_consensus.cc:348] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:30.678171 6383 raft_consensus.cc:374] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:30.678105 6384 consensus_queue.cc:260] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:30.678261 6383 raft_consensus.cc:729] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a1e9c0257c4138a28eb98fef8ffb22, State: Initialized, Role: FOLLOWER
I20260619 04:45:30.678469 6383 consensus_queue.cc:260] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:30.678635 6384 ts_tablet_manager.cc:1428] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 04:45:30.679668 6383 ts_tablet_manager.cc:1428] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:45:31.026628 6362 raft_consensus.cc:482] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:31.026847 6362 raft_consensus.cc:504] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.027571 6362 leader_election.cc:288] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929), 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043)
I20260619 04:45:31.028209 6089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f58e65f77342c896e35dd3124166b7" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edb4355763f47a6a52593b0315097e9" is_pre_election: true
I20260619 04:45:31.028203 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f58e65f77342c896e35dd3124166b7" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" is_pre_election: true
I20260619 04:45:31.028450 6089 raft_consensus.cc:2454] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 0.
I20260619 04:45:31.028502 5946 raft_consensus.cc:2454] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 0.
I20260619 04:45:31.028962 6156 leader_election.cc:302] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edb4355763f47a6a52593b0315097e9, dd3869da6ad648fc8bbcccd310136089; no voters:
I20260619 04:45:31.029242 6362 raft_consensus.cc:2781] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:31.029400 6362 raft_consensus.cc:482] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:31.029505 6362 raft_consensus.cc:3037] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:31.031821 6362 raft_consensus.cc:504] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.032402 6362 leader_election.cc:288] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 election: Requested vote from peers 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929), 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043)
I20260619 04:45:31.032807 5946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f58e65f77342c896e35dd3124166b7" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11a1e9c0257c4138a28eb98fef8ffb22"
I20260619 04:45:31.032867 6089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f58e65f77342c896e35dd3124166b7" candidate_uuid: "dd3869da6ad648fc8bbcccd310136089" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edb4355763f47a6a52593b0315097e9"
I20260619 04:45:31.033027 5946 raft_consensus.cc:3037] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:31.033121 6089 raft_consensus.cc:3037] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:31.035094 6089 raft_consensus.cc:2454] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 1.
I20260619 04:45:31.035104 5946 raft_consensus.cc:2454] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd3869da6ad648fc8bbcccd310136089 in term 1.
I20260619 04:45:31.035477 6156 leader_election.cc:302] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6edb4355763f47a6a52593b0315097e9, dd3869da6ad648fc8bbcccd310136089; no voters:
I20260619 04:45:31.035766 6362 raft_consensus.cc:2781] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:31.035907 6362 raft_consensus.cc:686] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 1 LEADER]: Becoming Leader. State: Replica: dd3869da6ad648fc8bbcccd310136089, State: Running, Role: LEADER
I20260619 04:45:31.036233 6362 consensus_queue.cc:237] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.038779 5788 catalog_manager.cc:5495] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd3869da6ad648fc8bbcccd310136089 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "dd3869da6ad648fc8bbcccd310136089" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } health_report { overall_health: HEALTHY } } }
I20260619 04:45:31.137219 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.132785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.3.243.131:39149)
0619 04:45:31.133167 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:31.133171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:31.133186 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:45:31.133235 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:31.133239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:31.133294 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:31.133372 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:31.133710 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.134429 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.135664 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.136038 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.136187 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.136548 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:31.136648 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:31.136712 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:31.136827 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 04:45:31.136880 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":214,"threads_started":1}
I20260619 04:45:31.144933 5946 raft_consensus.cc:1261] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Refusing update from remote peer dd3869da6ad648fc8bbcccd310136089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:31.144961 6089 raft_consensus.cc:1261] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Refusing update from remote peer dd3869da6ad648fc8bbcccd310136089: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:31.145535 6298 consensus_queue.cc:1035] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:31.145836 6362 consensus_queue.cc:1035] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:31.211460 5788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:31.211743 5788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:31.214434 5788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770: Sending DeleteTablet for 3 replicas of tablet 34f58e65f77342c896e35dd3124166b7
I20260619 04:45:31.215229 5926 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f58e65f77342c896e35dd3124166b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:31 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:41562
I20260619 04:45:31.215283 6064 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f58e65f77342c896e35dd3124166b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:31 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:54856
I20260619 04:45:31.215368 6204 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f58e65f77342c896e35dd3124166b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:31 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:49026
I20260619 04:45:31.216565 5788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:45:31.217379 5788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:31.225977 6065 tablet_service.cc:1467] Processing CreateTablet for tablet f50b23eac02643a38bee6e3d9930ad38 (DEFAULT_TABLE table=test-table [id=dd399f2a9bbb43e691fe55de59ee03d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:31.226432 6065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50b23eac02643a38bee6e3d9930ad38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:31.227978 6406 tablet_replica.cc:330] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: stopping tablet replica
I20260619 04:45:31.228391 6406 raft_consensus.cc:2229] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:31.228760 6406 raft_consensus.cc:2258] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:31.228945 6407 tablet_replica.cc:330] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: stopping tablet replica
I20260619 04:45:31.229071 5926 tablet_service.cc:1467] Processing CreateTablet for tablet f50b23eac02643a38bee6e3d9930ad38 (DEFAULT_TABLE table=test-table [id=dd399f2a9bbb43e691fe55de59ee03d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:31.229324 6203 tablet_service.cc:1467] Processing CreateTablet for tablet f50b23eac02643a38bee6e3d9930ad38 (DEFAULT_TABLE table=test-table [id=dd399f2a9bbb43e691fe55de59ee03d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:31.229444 6408 tablet_replica.cc:330] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: stopping tablet replica
I20260619 04:45:31.229604 5926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50b23eac02643a38bee6e3d9930ad38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:31.229578 6407 raft_consensus.cc:2229] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:31.229637 6408 raft_consensus.cc:2229] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:45:31.229756 6203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f50b23eac02643a38bee6e3d9930ad38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:31.229813 6407 raft_consensus.cc:2258] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:31.230010 6408 raft_consensus.cc:2258] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:31.230785 6406 ts_tablet_manager.cc:1905] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:31.230846 6407 ts_tablet_manager.cc:1905] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:31.237455 6407 ts_tablet_manager.cc:1918] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:45:31.237618 6407 log.cc:1196] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/34f58e65f77342c896e35dd3124166b7
I20260619 04:45:31.237680 6406 ts_tablet_manager.cc:1918] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:45:31.237831 6406 log.cc:1196] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/34f58e65f77342c896e35dd3124166b7
I20260619 04:45:31.238108 6407 ts_tablet_manager.cc:1939] T 34f58e65f77342c896e35dd3124166b7 P 6edb4355763f47a6a52593b0315097e9: Deleting consensus metadata
I20260619 04:45:31.238300 6406 ts_tablet_manager.cc:1939] T 34f58e65f77342c896e35dd3124166b7 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting consensus metadata
I20260619 04:45:31.239243 5774 catalog_manager.cc:4841] TS 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043): tablet 34f58e65f77342c896e35dd3124166b7 (table test-table [id=95b691794e9042a99a3ead9df35b4929]) successfully deleted
I20260619 04:45:31.239482 5775 catalog_manager.cc:4841] TS 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929): tablet 34f58e65f77342c896e35dd3124166b7 (table test-table [id=95b691794e9042a99a3ead9df35b4929]) successfully deleted
I20260619 04:45:31.240178 6408 ts_tablet_manager.cc:1905] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:31.242256 6409 tablet_bootstrap.cc:492] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: Bootstrap starting.
I20260619 04:45:31.244180 6409 tablet_bootstrap.cc:654] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:31.245777 6410 tablet_bootstrap.cc:492] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: Bootstrap starting.
I20260619 04:45:31.247422 6410 tablet_bootstrap.cc:654] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:31.248020 6408 ts_tablet_manager.cc:1918] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:45:31.248173 6408 log.cc:1196] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/34f58e65f77342c896e35dd3124166b7
I20260619 04:45:31.248760 6408 ts_tablet_manager.cc:1939] T 34f58e65f77342c896e35dd3124166b7 P dd3869da6ad648fc8bbcccd310136089: Deleting consensus metadata
I20260619 04:45:31.249927 5773 catalog_manager.cc:4841] TS dd3869da6ad648fc8bbcccd310136089 (127.3.243.131:39149): tablet 34f58e65f77342c896e35dd3124166b7 (table test-table [id=95b691794e9042a99a3ead9df35b4929]) successfully deleted
I20260619 04:45:31.250770 6409 tablet_bootstrap.cc:492] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: No bootstrap required, opened a new log
I20260619 04:45:31.250821 6410 tablet_bootstrap.cc:492] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: No bootstrap required, opened a new log
I20260619 04:45:31.250941 6409 ts_tablet_manager.cc:1397] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260619 04:45:31.251010 6410 ts_tablet_manager.cc:1397] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 04:45:31.251652 6411 tablet_bootstrap.cc:492] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: Bootstrap starting.
I20260619 04:45:31.251597 6409 raft_consensus.cc:348] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.251825 6409 raft_consensus.cc:374] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:31.251926 6409 raft_consensus.cc:729] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd3869da6ad648fc8bbcccd310136089, State: Initialized, Role: FOLLOWER
I20260619 04:45:31.251797 6410 raft_consensus.cc:348] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.252033 6410 raft_consensus.cc:374] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:31.252169 6410 raft_consensus.cc:729] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a1e9c0257c4138a28eb98fef8ffb22, State: Initialized, Role: FOLLOWER
I20260619 04:45:31.252197 6409 consensus_queue.cc:260] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.252435 6410 consensus_queue.cc:260] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.252813 6409 ts_tablet_manager.cc:1428] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:45:31.254114 6411 tablet_bootstrap.cc:654] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:31.254281 6410 ts_tablet_manager.cc:1428] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260619 04:45:31.256340 6411 tablet_bootstrap.cc:492] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: No bootstrap required, opened a new log
I20260619 04:45:31.256479 6411 ts_tablet_manager.cc:1397] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260619 04:45:31.257264 6411 raft_consensus.cc:348] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.257531 6411 raft_consensus.cc:374] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:31.257638 6411 raft_consensus.cc:729] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6edb4355763f47a6a52593b0315097e9, State: Initialized, Role: FOLLOWER
I20260619 04:45:31.257908 6411 consensus_queue.cc:260] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.258419 6411 ts_tablet_manager.cc:1428] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 04:45:31.319957 6388 raft_consensus.cc:482] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:31.320240 6388 raft_consensus.cc:504] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.321717 6388 leader_election.cc:288] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043), dd3869da6ad648fc8bbcccd310136089 (127.3.243.131:39149)
I20260619 04:45:31.329639 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.322149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:38575 (local address 127.3.243.131:39149)
0619 04:45:31.322356 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:31.322365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:31.323352 (+ 987us) server_negotiation.cc:407] Connection header received
0619 04:45:31.323439 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:31.323444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:31.323513 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:31.323617 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:31.324496 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.325652 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.327818 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.328687 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.328890 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.328973 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:31.329262 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 04:45:31.329392 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:45:31.329828 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.322022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:39149 (local address 127.3.243.129:38575)
0619 04:45:31.323235 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:31.323250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:31.323354 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:31.323649 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:31.323654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:31.323698 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:31.324299 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:31.324311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.325891 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.325896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:31.327544 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:31.327556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.328934 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.328942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:31.329100 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.329127 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 04:45:31.329237 (+ 110us) client_negotiation.cc:240] Negotiation successful
0619 04:45:31.329363 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1113,"mutex_wait_us":59,"spinlock_wait_cycles":896,"thread_start_us":932,"threads_started":1}
I20260619 04:45:31.329887 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.322022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:35043 (local address 127.3.243.129:59569)
0619 04:45:31.323074 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:31.323097 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:31.323273 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:31.323649 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:31.323654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:31.323704 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:31.324199 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:31.324216 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.325740 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.325745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:31.327385 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:31.327397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.329012 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.329021 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:31.329178 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.329200 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 04:45:31.329303 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 04:45:31.329438 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"mutex_wait_us":24,"spinlock_wait_cycles":3328,"thread_start_us":711,"threads_started":1}
I20260619 04:45:31.330039 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.322142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:59569 (local address 127.3.243.130:35043)
0619 04:45:31.322817 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:31.322825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:31.323213 (+ 388us) server_negotiation.cc:407] Connection header received
0619 04:45:31.323430 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:31.323435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:31.323513 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:31.323617 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:31.324466 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.325595 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.327555 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.328706 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.328919 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.329001 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:31.329338 (+ 337us) server_negotiation.cc:299] Negotiation successful
0619 04:45:31.329463 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":387,"threads_started":1}
I20260619 04:45:31.330446 6228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50b23eac02643a38bee6e3d9930ad38" candidate_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3869da6ad648fc8bbcccd310136089" is_pre_election: true
I20260619 04:45:31.330529 6089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50b23eac02643a38bee6e3d9930ad38" candidate_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edb4355763f47a6a52593b0315097e9" is_pre_election: true
I20260619 04:45:31.330734 6089 raft_consensus.cc:2454] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11a1e9c0257c4138a28eb98fef8ffb22 in term 0.
I20260619 04:45:31.330715 6228 raft_consensus.cc:2454] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11a1e9c0257c4138a28eb98fef8ffb22 in term 0.
I20260619 04:45:31.331187 5877 leader_election.cc:302] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11a1e9c0257c4138a28eb98fef8ffb22, 6edb4355763f47a6a52593b0315097e9; no voters:
I20260619 04:45:31.331626 6388 raft_consensus.cc:2781] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:31.331728 6388 raft_consensus.cc:482] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:31.331871 6388 raft_consensus.cc:3037] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:31.333959 6388 raft_consensus.cc:504] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.334556 6388 leader_election.cc:288] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [CANDIDATE]: Term 1 election: Requested vote from peers 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043), dd3869da6ad648fc8bbcccd310136089 (127.3.243.131:39149)
I20260619 04:45:31.335119 6089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50b23eac02643a38bee6e3d9930ad38" candidate_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6edb4355763f47a6a52593b0315097e9"
I20260619 04:45:31.335119 6228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f50b23eac02643a38bee6e3d9930ad38" candidate_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd3869da6ad648fc8bbcccd310136089"
I20260619 04:45:31.335337 6228 raft_consensus.cc:3037] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:31.335337 6089 raft_consensus.cc:3037] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:31.338223 6228 raft_consensus.cc:2454] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11a1e9c0257c4138a28eb98fef8ffb22 in term 1.
I20260619 04:45:31.338223 6089 raft_consensus.cc:2454] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11a1e9c0257c4138a28eb98fef8ffb22 in term 1.
I20260619 04:45:31.338727 5876 leader_election.cc:302] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11a1e9c0257c4138a28eb98fef8ffb22, 6edb4355763f47a6a52593b0315097e9, dd3869da6ad648fc8bbcccd310136089; no voters:
I20260619 04:45:31.339089 6388 raft_consensus.cc:2781] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:31.339509 6388 raft_consensus.cc:686] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 LEADER]: Becoming Leader. State: Replica: 11a1e9c0257c4138a28eb98fef8ffb22, State: Running, Role: LEADER
I20260619 04:45:31.339876 6388 consensus_queue.cc:237] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } }
I20260619 04:45:31.343322 5787 catalog_manager.cc:5495] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129). New cstate: current_term: 1 leader_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11a1e9c0257c4138a28eb98fef8ffb22" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 } health_report { overall_health: UNKNOWN } } }
I20260619 04:45:31.366968 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.361149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.3.243.129:41929)
0619 04:45:31.361786 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:31.361796 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:31.361815 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:45:31.361894 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:31.361897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:31.361990 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:31.362125 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:31.362550 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.363287 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.364750 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.365210 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.365301 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.365723 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:31.366372 (+ 649us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:31.366477 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:31.366578 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 04:45:31.366693 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":377,"threads_started":1}
I20260619 04:45:31.381091 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.376242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:48215 (local address 127.3.243.130:35043)
0619 04:45:31.376443 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:31.376448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:31.377063 (+ 615us) server_negotiation.cc:407] Connection header received
0619 04:45:31.377200 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:31.377204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:31.377265 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:31.377342 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:31.377741 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.378382 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.379643 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.380007 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.380096 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.380546 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:31.380661 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:31.380727 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:31.380850 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 04:45:31.380906 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:45:31.381210 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.376292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:35043 (local address 127.3.243.129:48215)
0619 04:45:31.376979 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:31.376994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:31.377101 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:31.377365 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:31.377368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:31.377376 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:45:31.377626 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:31.377632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.378523 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.378526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:31.379533 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:31.379543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.380219 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.380224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:31.380349 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.380396 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:45:31.380744 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:45:31.380757 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 04:45:31.380814 (+ 57us) client_negotiation.cc:240] Negotiation successful
0619 04:45:31.380907 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":353,"threads_started":1}
I20260619 04:45:31.389281 6228 raft_consensus.cc:1261] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Refusing update from remote peer 11a1e9c0257c4138a28eb98fef8ffb22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:31.389281 6089 raft_consensus.cc:1261] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Refusing update from remote peer 11a1e9c0257c4138a28eb98fef8ffb22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:31.389953 6418 consensus_queue.cc:1035] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd3869da6ad648fc8bbcccd310136089" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:31.390341 6388 consensus_queue.cc:1035] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6edb4355763f47a6a52593b0315097e9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:31.415577 6420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.410058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:48733 (local address 127.3.243.129:41929)
0619 04:45:31.410286 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:31.410292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:31.410727 (+ 435us) server_negotiation.cc:407] Connection header received
0619 04:45:31.410882 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:31.410885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:31.410948 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:31.411018 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:31.411480 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.412464 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.414209 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.414609 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.414700 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.415017 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:31.415107 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:31.415172 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:31.415307 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 04:45:31.415363 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 04:45:31.415776 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:31.409984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41929 (local address 127.3.243.130:48733)
0619 04:45:31.410630 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:31.410646 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:31.410748 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:31.411086 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:31.411092 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:31.411104 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:45:31.411308 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:31.411315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.412600 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.412605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:31.414088 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:31.414099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:31.414736 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:31.414740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:31.414854 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:31.414888 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:45:31.415189 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:45:31.415196 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 04:45:31.415297 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 04:45:31.415419 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":358,"threads_started":1}
W20260619 04:45:31.416992 6393 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 04:45:33.009657 5787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:33.009863 5787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40940:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:33.013720 5787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d51536b0f564b9787ca91433517e770: Sending DeleteTablet for 3 replicas of tablet f50b23eac02643a38bee6e3d9930ad38
I20260619 04:45:33.014463 6065 tablet_service.cc:1514] Processing DeleteTablet for tablet f50b23eac02643a38bee6e3d9930ad38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:33 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:54856
I20260619 04:45:33.014465 5926 tablet_service.cc:1514] Processing DeleteTablet for tablet f50b23eac02643a38bee6e3d9930ad38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:33 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:41562
I20260619 04:45:33.014626 6203 tablet_service.cc:1514] Processing DeleteTablet for tablet f50b23eac02643a38bee6e3d9930ad38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:33 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:49026
I20260619 04:45:33.016343 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5860
I20260619 04:45:33.027293 6452 tablet_replica.cc:330] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: stopping tablet replica
I20260619 04:45:33.027480 6451 tablet_replica.cc:330] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: stopping tablet replica
I20260619 04:45:33.027675 6451 raft_consensus.cc:2229] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:45:33.027745 6452 raft_consensus.cc:2229] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:33.028080 6452 raft_consensus.cc:2258] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:33.028290 6451 raft_consensus.cc:2258] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:33.028808 6453 tablet_replica.cc:330] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: stopping tablet replica
I20260619 04:45:33.029065 6453 raft_consensus.cc:2229] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:33.029335 6452 ts_tablet_manager.cc:1905] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:33.029351 6453 raft_consensus.cc:2258] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:33.031076 6453 ts_tablet_manager.cc:1905] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:33.031766 6451 ts_tablet_manager.cc:1905] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:33.035876 6452 ts_tablet_manager.cc:1918] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:45:33.036065 6452 log.cc:1196] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/f50b23eac02643a38bee6e3d9930ad38
I20260619 04:45:33.036576 6452 ts_tablet_manager.cc:1939] T f50b23eac02643a38bee6e3d9930ad38 P 6edb4355763f47a6a52593b0315097e9: Deleting consensus metadata
I20260619 04:45:33.037734 5774 catalog_manager.cc:4841] TS 6edb4355763f47a6a52593b0315097e9 (127.3.243.130:35043): tablet f50b23eac02643a38bee6e3d9930ad38 (table test-table [id=dd399f2a9bbb43e691fe55de59ee03d3]) successfully deleted
I20260619 04:45:33.038195 6453 ts_tablet_manager.cc:1918] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:45:33.038345 6453 log.cc:1196] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/f50b23eac02643a38bee6e3d9930ad38
I20260619 04:45:33.038745 6453 ts_tablet_manager.cc:1939] T f50b23eac02643a38bee6e3d9930ad38 P dd3869da6ad648fc8bbcccd310136089: Deleting consensus metadata
I20260619 04:45:33.039664 5773 catalog_manager.cc:4841] TS dd3869da6ad648fc8bbcccd310136089 (127.3.243.131:39149): tablet f50b23eac02643a38bee6e3d9930ad38 (table test-table [id=dd399f2a9bbb43e691fe55de59ee03d3]) successfully deleted
I20260619 04:45:33.041407 6451 ts_tablet_manager.cc:1918] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 04:45:33.041546 6451 log.cc:1196] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/f50b23eac02643a38bee6e3d9930ad38
I20260619 04:45:33.041975 6451 ts_tablet_manager.cc:1939] T f50b23eac02643a38bee6e3d9930ad38 P 11a1e9c0257c4138a28eb98fef8ffb22: Deleting consensus metadata
I20260619 04:45:33.043095 5775 catalog_manager.cc:4841] TS 11a1e9c0257c4138a28eb98fef8ffb22 (127.3.243.129:41929): tablet f50b23eac02643a38bee6e3d9930ad38 (table test-table [id=dd399f2a9bbb43e691fe55de59ee03d3]) successfully deleted
Jun 19 04:45:33 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:45:33.050307 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:33.018030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.3.243.129:41929)
0619 04:45:33.034856 (+ 16826us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:33.034863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:33.034886 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:45:33.034942 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:33.034947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:33.035021 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:33.035135 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:33.036955 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:33.038121 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:33.039973 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:33.040495 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:33.040592 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:33.045139 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:33.045152 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:33.045169 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:33.045200 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:33.048000 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:33.048650 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:33.048658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:33.048661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:33.048799 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:33.049112 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:33.049115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:33.049117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:33.049437 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:33.049661 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:33.049892 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 04:45:33.050059 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16681,"thread_start_us":16791,"threads_started":1}
W20260619 04:45:33.050948 5991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41592
W20260619 04:45:33.051462 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5860: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:45:33.051676 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 5860
I20260619 04:45:33.074159 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6000
Jun 19 04:45:33 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:45:33.092213 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:33.074901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.3.243.130:35043)
0619 04:45:33.075426 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:33.075431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:33.075447 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:45:33.075502 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:33.075506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:33.075565 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:33.075654 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:33.076947 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:33.077665 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:33.078823 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:33.079218 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:33.079314 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:33.087125 (+ 7811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:33.087137 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:33.087153 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:33.087189 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:33.089818 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:33.090387 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:33.090394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:33.090396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:33.090529 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:33.090834 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:33.090837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:33.090839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:33.091170 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:33.091388 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:33.091603 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 04:45:33.091762 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":290,"threads_started":1}
W20260619 04:45:33.092864 6130 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54870
W20260619 04:45:33.093478 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6000: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:45:33.093640 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 6000
I20260619 04:45:33.114835 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6139
Jun 19 04:45:33 dist-test-slave-2qp8 krb5kdc[5744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844330, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:45:33.129280 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:33.115591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.3.243.131:39149)
0619 04:45:33.116065 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:33.116070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:33.116085 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:45:33.116204 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:33.116207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:33.116273 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:33.116362 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:33.117592 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:33.118303 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:33.119454 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:33.119843 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:33.119933 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:33.124070 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:33.124095 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:33.124152 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:33.124193 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:33.126966 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:33.127547 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:33.127553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:33.127555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:33.127663 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:33.128015 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:33.128018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:33.128020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:33.128374 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:33.128598 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:33.128833 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 04:45:33.128996 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":276,"threads_started":1}
W20260619 04:45:33.129945 6269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49046
W20260619 04:45:33.130541 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6139: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:45:33.130715 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 6139
I20260619 04:45:33.152283 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 5757
I20260619 04:45:33.153445 5829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:33.269953 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 5757
2026-06-19T04:45:33Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8447 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:33 dist-test-slave-2qp8 krb5kdc[6461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:33 dist-test-slave-2qp8 krb5kdc[6461](info): set up 2 sockets
Jun 19 04:45:33 dist-test-slave-2qp8 krb5kdc[6461](info): commencing operation
krb5kdc: starting...
W20260619 04:45:34.368474 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:34 dist-test-slave-2qp8 krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:34.559029 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:37891
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:34.868045 6474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:34.868381 6474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:34.868549 6474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:34.875473 6474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:34.875626 6474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:34.875717 6474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:34.875828 6474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:34.875898 6474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:34.886443 6474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:37891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:34.887709 6474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:34.889073 6474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:34.900743 6479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:34.900755 6482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:34.900771 6480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:34.902165 6474 server_base.cc:1034] running on GCE node
I20260619 04:45:34.902910 6474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:34.904398 6474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:34.905640 6474 hybrid_clock.cc:648] HybridClock initialized: now 1781844334905586 us; error 40 us; skew 500 ppm
I20260619 04:45:34.905969 6474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:34 dist-test-slave-2qp8 krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:34.908914 6474 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:34.910418 6474 webserver.cc:460] Webserver started at http://127.3.243.190:35671/ using document root <none> and password file <none>
I20260619 04:45:34.911087 6474 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:34.911226 6474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:34.911537 6474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:34.914559 6474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "0f6c234ffbc249b894b9628119056cb8"
format_stamp: "Formatted at 2026-06-19 04:45:34 on dist-test-slave-2qp8"
server_key: "db2288b90c6a737fa2ba95c8e06bc5fb"
server_key_iv: "6420788eed1f7c2b742754136c472a9e"
server_key_version: "encryptionkey@0"
I20260619 04:45:34.915314 6474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "0f6c234ffbc249b894b9628119056cb8"
format_stamp: "Formatted at 2026-06-19 04:45:34 on dist-test-slave-2qp8"
server_key: "db2288b90c6a737fa2ba95c8e06bc5fb"
server_key_iv: "6420788eed1f7c2b742754136c472a9e"
server_key_version: "encryptionkey@0"
I20260619 04:45:34.920256 6474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 04:45:34.923385 6489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:34.924731 6474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:34.924901 6474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "0f6c234ffbc249b894b9628119056cb8"
format_stamp: "Formatted at 2026-06-19 04:45:34 on dist-test-slave-2qp8"
server_key: "db2288b90c6a737fa2ba95c8e06bc5fb"
server_key_iv: "6420788eed1f7c2b742754136c472a9e"
server_key_version: "encryptionkey@0"
I20260619 04:45:34.925087 6474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:34.942147 6474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:34.946074 6474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:34.946360 6474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:34.964200 6474 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:37891
I20260619 04:45:34.964190 6540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:37891 every 8 connection(s)
I20260619 04:45:34.965564 6474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:34.969275 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:34.975849 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 6474
I20260619 04:45:34.976105 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:34.976934 4046 external_mini_cluster.cc:1428] Setting key f108a293264059558890bfe2ca41efd1
I20260619 04:45:34.980372 6541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8: Bootstrap starting.
I20260619 04:45:34.983907 6541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:34.985061 6541 log.cc:824] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:34.987939 6541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8: No bootstrap required, opened a new log
Jun 19 04:45:34 dist-test-slave-2qp8 krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:34.996867 6541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c234ffbc249b894b9628119056cb8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37891 } }
I20260619 04:45:34.997264 6541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:34.997395 6541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6c234ffbc249b894b9628119056cb8, State: Initialized, Role: FOLLOWER
I20260619 04:45:34.998020 6541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c234ffbc249b894b9628119056cb8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37891 } }
I20260619 04:45:34.998265 6541 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:34.998421 6541 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:34.998615 6541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:34.999869 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:34.978917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.3.243.190:37891)
0619 04:45:34.979684 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:34.979696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:34.979735 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:45:34.981500 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:34.981522 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:34.981893 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:34.982217 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:34.984044 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:34.985441 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:34.987084 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:34.987708 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:34.987828 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:34.992544 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:34.992552 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:34.992569 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:34.992614 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:34.995747 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:34.996485 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:34.996492 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:34.996498 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:34.996647 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:34.997013 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:34.997020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:34.997023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:34.997677 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:34.998002 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:34.998428 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 04:45:34.998857 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":364,"threads_started":1}
I20260619 04:45:35.000982 6541 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c234ffbc249b894b9628119056cb8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37891 } }
I20260619 04:45:35.001474 6541 leader_election.cc:302] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f6c234ffbc249b894b9628119056cb8; no voters:
I20260619 04:45:35.002131 6541 leader_election.cc:288] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:35.002249 6546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:35.003343 6546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6c234ffbc249b894b9628119056cb8, State: Running, Role: LEADER
I20260619 04:45:35.003883 6546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c234ffbc249b894b9628119056cb8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37891 } }
I20260619 04:45:35.004608 6541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:35.006728 6547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f6c234ffbc249b894b9628119056cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c234ffbc249b894b9628119056cb8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37891 } } }
I20260619 04:45:35.006925 6548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f6c234ffbc249b894b9628119056cb8. Latest consensus state: current_term: 1 leader_uuid: "0f6c234ffbc249b894b9628119056cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f6c234ffbc249b894b9628119056cb8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37891 } } }
I20260619 04:45:35.007234 6548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:35.007241 6547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:35.013195 6553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:35.018594 6553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:35.027081 6553 catalog_manager.cc:1347] Generated new cluster ID: 15f80b4b8fb74c709f9a4f8974d7b9b4
I20260619 04:45:35.027204 6553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:35.047359 6553 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:35.048959 6553 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:35.059679 6553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f6c234ffbc249b894b9628119056cb8: Generated new TSK 0
I20260619 04:45:35.060554 6553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:45:35.119050 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37891
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:35.450469 6567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:35.450757 6567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:35.450852 6567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:35.457978 6567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:35.458087 6567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:35.458199 6567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:35.468091 6567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:35.469417 6567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:35.470662 6567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:35.483759 6575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:35.483800 6573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:35.483807 6572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:35.484556 6567 server_base.cc:1034] running on GCE node
I20260619 04:45:35.485237 6567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:35.486173 6567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:35.487453 6567 hybrid_clock.cc:648] HybridClock initialized: now 1781844335487390 us; error 58 us; skew 500 ppm
I20260619 04:45:35.487780 6567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:35 dist-test-slave-2qp8 krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:35.490932 6567 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:45:35.493016 6567 webserver.cc:460] Webserver started at http://127.3.243.129:35309/ using document root <none> and password file <none>
I20260619 04:45:35.493722 6567 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:35.493862 6567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:35.494187 6567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:35.497208 6567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "d6323aec6a65408a938215c0841999bf"
format_stamp: "Formatted at 2026-06-19 04:45:35 on dist-test-slave-2qp8"
server_key: "afae41d0f2493e97b645bc68f0edaccb"
server_key_iv: "156a931a4922ae8b3933d33f985ed1d6"
server_key_version: "encryptionkey@0"
I20260619 04:45:35.497982 6567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "d6323aec6a65408a938215c0841999bf"
format_stamp: "Formatted at 2026-06-19 04:45:35 on dist-test-slave-2qp8"
server_key: "afae41d0f2493e97b645bc68f0edaccb"
server_key_iv: "156a931a4922ae8b3933d33f985ed1d6"
server_key_version: "encryptionkey@0"
I20260619 04:45:35.503290 6567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:45:35.507055 6582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:35.508420 6567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:35.508595 6567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "d6323aec6a65408a938215c0841999bf"
format_stamp: "Formatted at 2026-06-19 04:45:35 on dist-test-slave-2qp8"
server_key: "afae41d0f2493e97b645bc68f0edaccb"
server_key_iv: "156a931a4922ae8b3933d33f985ed1d6"
server_key_version: "encryptionkey@0"
I20260619 04:45:35.508805 6567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:35.524523 6567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:35.528528 6567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:35.528992 6567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:35.530129 6567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:35.531438 6567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:35.531541 6567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:35.531704 6567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:35.531780 6567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:35.564531 6567 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:39819
I20260619 04:45:35.564580 6694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:39819 every 8 connection(s)
I20260619 04:45:35.565853 6567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:35.569885 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 6567
I20260619 04:45:35.570114 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:35.570925 4046 external_mini_cluster.cc:1428] Setting key 85846bfad86314bd9c6f9642dac786e1
Jun 19 04:45:35 dist-test-slave-2qp8 krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:35.587435 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:35.568404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:44537 (local address 127.3.243.190:37891)
0619 04:45:35.568803 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:35.568807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:35.570123 (+ 1316us) server_negotiation.cc:407] Connection header received
0619 04:45:35.572539 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:35.572543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:35.572603 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:35.572686 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:35.574260 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:35.575425 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:35.576953 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:35.577648 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:35.577787 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:35.581687 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:35.581692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:35.581695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:35.581727 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:35.584394 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:35.585165 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:35.585170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:35.585176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:35.585298 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:35.585735 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:35.585740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:35.585743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:35.586010 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:35.586163 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:35.586857 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 04:45:35.587075 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":212,"threads_started":1}
I20260619 04:45:35.588531 6698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:35.569088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37891 (local address 127.3.243.129:44537)
0619 04:45:35.569928 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:35.569979 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:35.572232 (+ 2253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:35.572991 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:35.573010 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:35.573552 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:35.574089 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:35.574104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:35.575586 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:35.575589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:35.576810 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:35.576822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:35.577842 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:35.577846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:35.578039 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:35.578762 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:35.578797 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:35.581466 (+ 2669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:35.584572 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:35.584582 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:35.584599 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:35.585036 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:35.585419 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:35.585423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:35.585427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:35.585617 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:35.586136 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:35.586144 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:35.586494 (+ 350us) client_negotiation.cc:769] Sending connection context
0619 04:45:35.586755 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 04:45:35.587298 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":318,"threads_started":1}
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:45:35.590128 6695 heartbeater.cc:347] Connected to a master server at 127.3.243.190:37891
I20260619 04:45:35.590436 6695 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
I20260619 04:45:35.591346 6695 heartbeater.cc:511] Master 127.3.243.190:37891 requested a full tablet report, sending...
I20260619 04:45:35.593578 6506 ts_manager.cc:194] Registered new tserver with Master: d6323aec6a65408a938215c0841999bf (127.3.243.129:39819)
I20260619 04:45:35.595005 6506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:44537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:45:35.615505 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37891
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:35.934248 6701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:35.934526 6701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:35.934620 6701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:35.941532 6701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:35.941629 6701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:35.941737 6701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:35.951130 6701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:35.952327 6701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:35.953541 6701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:35.966138 6706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:35.966390 6709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:35.966842 6707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:35.967299 6701 server_base.cc:1034] running on GCE node
I20260619 04:45:35.968042 6701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:35.969012 6701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:35.970239 6701 hybrid_clock.cc:648] HybridClock initialized: now 1781844335970178 us; error 56 us; skew 500 ppm
I20260619 04:45:35.970561 6701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:35 dist-test-slave-2qp8 krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:35.973811 6701 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:45:35.975401 6701 webserver.cc:460] Webserver started at http://127.3.243.130:36471/ using document root <none> and password file <none>
I20260619 04:45:35.976074 6701 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:35.976262 6701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:35.976560 6701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:35.979346 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "15d7936fcf8a412d9efaec7867e202c9"
format_stamp: "Formatted at 2026-06-19 04:45:35 on dist-test-slave-2qp8"
server_key: "c48417e2bed646d2b3f97f717e2a318b"
server_key_iv: "c21508e6d62d078eabf50618465bc7b5"
server_key_version: "encryptionkey@0"
I20260619 04:45:35.980171 6701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "15d7936fcf8a412d9efaec7867e202c9"
format_stamp: "Formatted at 2026-06-19 04:45:35 on dist-test-slave-2qp8"
server_key: "c48417e2bed646d2b3f97f717e2a318b"
server_key_iv: "c21508e6d62d078eabf50618465bc7b5"
server_key_version: "encryptionkey@0"
I20260619 04:45:35.984988 6701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:45:35.988332 6716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:35.989563 6701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:35.989734 6701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "15d7936fcf8a412d9efaec7867e202c9"
format_stamp: "Formatted at 2026-06-19 04:45:35 on dist-test-slave-2qp8"
server_key: "c48417e2bed646d2b3f97f717e2a318b"
server_key_iv: "c21508e6d62d078eabf50618465bc7b5"
server_key_version: "encryptionkey@0"
I20260619 04:45:35.989923 6701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:36.014241 6701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:36.018636 6701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:36.018935 6701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:36.019922 6701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:36.021270 6701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:36.021373 6701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:36.021512 6701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:36.021582 6701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:36.054020 6701 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:35047
I20260619 04:45:36.054071 6828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:35047 every 8 connection(s)
I20260619 04:45:36.055395 6701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:36.062628 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 6701
I20260619 04:45:36.062824 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:36.063242 4046 external_mini_cluster.cc:1428] Setting key eeae3dc894fc6cf899d3555b54001ba1
Jun 19 04:45:36 dist-test-slave-2qp8 krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:36.076040 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:36.057941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:34067 (local address 127.3.243.190:37891)
0619 04:45:36.058113 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:36.058117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:36.059283 (+ 1166us) server_negotiation.cc:407] Connection header received
0619 04:45:36.061335 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:36.061340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:36.061433 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:36.061552 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:36.063122 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:36.063864 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:36.065610 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:36.066352 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:36.066509 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:36.070210 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:36.070219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:36.070227 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:36.070273 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:36.072931 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:36.073854 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:36.073858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:36.073860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:36.073980 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:36.074433 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:36.074436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:36.074438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:36.074682 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:36.074783 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:36.075648 (+ 865us) server_negotiation.cc:299] Negotiation successful
0619 04:45:36.075812 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:45:36.077618 6831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:36.058439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37891 (local address 127.3.243.130:34067)
0619 04:45:36.059120 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:36.059156 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:36.061061 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:36.061785 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:36.061792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:36.062319 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:36.062955 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:36.062980 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:36.064039 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:36.064044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:36.065471 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:36.065483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:36.066554 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:36.066558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:36.066744 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:36.067429 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:36.067451 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:36.069970 (+ 2519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:36.073067 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:36.073078 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:36.073099 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:36.073739 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:36.074083 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:36.074090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:36.074097 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:36.074328 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:36.074809 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:36.074819 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:36.075259 (+ 440us) client_negotiation.cc:769] Sending connection context
0619 04:45:36.075568 (+ 309us) client_negotiation.cc:240] Negotiation successful
0619 04:45:36.076153 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":256,"threads_started":1}
I20260619 04:45:36.079020 6829 heartbeater.cc:347] Connected to a master server at 127.3.243.190:37891
I20260619 04:45:36.079325 6829 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:36.079999 6829 heartbeater.cc:511] Master 127.3.243.190:37891 requested a full tablet report, sending...
I20260619 04:45:36.081369 6506 ts_manager.cc:194] Registered new tserver with Master: 15d7936fcf8a412d9efaec7867e202c9 (127.3.243.130:35047)
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:45:36.082208 6506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:34067
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:45:36.107309 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37891
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:36.426548 6834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:36.426813 6834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:36.426904 6834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:36.434289 6834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:36.434399 6834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:36.434520 6834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:45:36.444171 6834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:36.445433 6834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:36.446748 6834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:36.459848 6839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:36.459914 6842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:36.459889 6840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:36.460543 6834 server_base.cc:1034] running on GCE node
I20260619 04:45:36.461156 6834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:36.461974 6834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:36.463177 6834 hybrid_clock.cc:648] HybridClock initialized: now 1781844336463131 us; error 46 us; skew 500 ppm
I20260619 04:45:36.463480 6834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:36 dist-test-slave-2qp8 krb5kdc[6461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:36.466594 6834 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:45:36.467948 6834 webserver.cc:460] Webserver started at http://127.3.243.131:45589/ using document root <none> and password file <none>
I20260619 04:45:36.468672 6834 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:36.468835 6834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:36.469180 6834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:36.471946 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "e6aa62b6893e455d9bc4ca0347a41335"
format_stamp: "Formatted at 2026-06-19 04:45:36 on dist-test-slave-2qp8"
server_key: "8bb2ed3071b35c8b3ff7dfa8dc6bb540"
server_key_iv: "12a1255df0ae37e561b80a97e8593e35"
server_key_version: "encryptionkey@0"
I20260619 04:45:36.472782 6834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "e6aa62b6893e455d9bc4ca0347a41335"
format_stamp: "Formatted at 2026-06-19 04:45:36 on dist-test-slave-2qp8"
server_key: "8bb2ed3071b35c8b3ff7dfa8dc6bb540"
server_key_iv: "12a1255df0ae37e561b80a97e8593e35"
server_key_version: "encryptionkey@0"
I20260619 04:45:36.477742 6834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260619 04:45:36.481108 6849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:36.482360 6834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:45:36.482527 6834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "e6aa62b6893e455d9bc4ca0347a41335"
format_stamp: "Formatted at 2026-06-19 04:45:36 on dist-test-slave-2qp8"
server_key: "8bb2ed3071b35c8b3ff7dfa8dc6bb540"
server_key_iv: "12a1255df0ae37e561b80a97e8593e35"
server_key_version: "encryptionkey@0"
I20260619 04:45:36.482725 6834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:36.494704 6834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:36.498791 6834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:36.499078 6834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:36.499980 6834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:36.501308 6834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:36.501410 6834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:36.501549 6834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:36.501619 6834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:36.534588 6834 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:41603
I20260619 04:45:36.534642 6961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:41603 every 8 connection(s)
I20260619 04:45:36.535881 6834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:45:36.544482 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 6834
I20260619 04:45:36.544680 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:45:36.545123 4046 external_mini_cluster.cc:1428] Setting key a198c71a5b9976a115ddf582f6419f6a
Jun 19 04:45:36 dist-test-slave-2qp8 krb5kdc[6461](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844336, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:36.554973 6697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:36.538419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:45725 (local address 127.3.243.190:37891)
0619 04:45:36.538613 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:36.538617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:36.539741 (+ 1124us) server_negotiation.cc:407] Connection header received
0619 04:45:36.541587 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:36.541593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:36.541696 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:36.541805 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:36.543241 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:36.543991 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:36.545682 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:36.546192 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:36.546325 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:36.549819 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:36.549824 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:36.549827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:36.549856 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:36.552093 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:36.552831 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:36.552836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:36.552840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:36.552970 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:36.553383 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:36.553390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:36.553398 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:36.553636 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:36.553798 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:36.554417 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 04:45:36.554632 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:45:36.556252 6964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:36.538881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37891 (local address 127.3.243.131:45725)
0619 04:45:36.539576 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:36.539613 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:36.541322 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:36.542034 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:36.542045 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:36.542536 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:36.543076 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:36.543093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:36.544179 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:36.544185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:36.545528 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:36.545547 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:36.546425 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:36.546432 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:36.546657 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:36.547328 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:36.547355 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:36.549630 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:36.552278 (+ 2648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:36.552286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:36.552300 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:36.552706 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:36.553084 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:36.553091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:36.553095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:36.553267 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:36.553760 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:36.553767 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:36.554077 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 04:45:36.554320 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 04:45:36.554818 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":251,"threads_started":1}
I20260619 04:45:36.557533 6962 heartbeater.cc:347] Connected to a master server at 127.3.243.190:37891
I20260619 04:45:36.557840 6962 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:36.558601 6962 heartbeater.cc:511] Master 127.3.243.190:37891 requested a full tablet report, sending...
I20260619 04:45:36.560187 6505 ts_manager.cc:194] Registered new tserver with Master: e6aa62b6893e455d9bc4ca0347a41335 (127.3.243.131:41603)
I20260619 04:45:36.561053 6505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:45725
I20260619 04:45:36.572350 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:45:36.597999 6695 heartbeater.cc:503] Master 127.3.243.190:37891 was elected leader, sending a full tablet report...
W20260619 04:45:36.603194 6697 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:45:36.600842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.3.243.190:37891)
0619 04:45:36.601086 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:36.601095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:36.601478 (+ 383us) server_negotiation.cc:407] Connection header received
0619 04:45:36.601612 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:36.601616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:36.601675 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:36.601756 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:36.602933 (+ 1177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51506: BlockingRecv error: recv got EOF from 127.0.0.1:51506 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 04:45:36.606698 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6567
W20260619 04:45:36.609588 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6567: Not authorized: Client connection negotiation failed: client connection to 127.3.243.129:39819: server requires authentication, but client does not have Kerberos credentials available
I20260619 04:45:36.609761 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 6567
I20260619 04:45:36.626067 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6701
W20260619 04:45:36.628896 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6701: Not authorized: Client connection negotiation failed: client connection to 127.3.243.130:35047: server requires authentication, but client does not have Kerberos credentials available
I20260619 04:45:36.629066 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 6701
I20260619 04:45:36.643949 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6834
W20260619 04:45:36.646869 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6834: Not authorized: Client connection negotiation failed: client connection to 127.3.243.131:41603: server requires authentication, but client does not have Kerberos credentials available
I20260619 04:45:36.647004 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 6834
I20260619 04:45:36.662132 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6474
I20260619 04:45:36.663229 6536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:36.764744 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 6474
2026-06-19T04:45:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3491 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:36 dist-test-slave-2qp8 krb5kdc[6979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:36 dist-test-slave-2qp8 krb5kdc[6979](info): set up 2 sockets
Jun 19 04:45:36 dist-test-slave-2qp8 krb5kdc[6979](info): commencing operation
krb5kdc: starting...
W20260619 04:45:37.840550 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:37 dist-test-slave-2qp8 krb5kdc[6979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:38.036094 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:38155
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:35299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:38155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:38.370776 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:38.371119 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:38.371273 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:38.378571 6992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:38.378716 6992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:38.378832 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:38.378928 6992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:38.379017 6992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:38.388904 6992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:35299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:38155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:38155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:38.390216 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:38.391521 6992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:38.403244 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:38.403395 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:38.403398 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:38.404350 6992 server_base.cc:1034] running on GCE node
I20260619 04:45:38.405184 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:38.406556 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:38.407793 6992 hybrid_clock.cc:648] HybridClock initialized: now 1781844338407747 us; error 32 us; skew 500 ppm
I20260619 04:45:38.408215 6992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:38 dist-test-slave-2qp8 krb5kdc[6979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844338, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:38.411286 6992 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:38.412835 6992 webserver.cc:460] Webserver started at http://127.3.243.190:38547/ using document root <none> and password file <none>
I20260619 04:45:38.413512 6992 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:38.413655 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:38.413959 6992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:38.417009 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "1a606aa97bb04098a9d7d4c3b2e9285a"
format_stamp: "Formatted at 2026-06-19 04:45:38 on dist-test-slave-2qp8"
server_key: "d19077f0acf1b0890cd56656844127c9"
server_key_iv: "8199a77c426ec5dd7bd95e4c459bc6a4"
server_key_version: "encryptionkey@0"
I20260619 04:45:38.417809 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "1a606aa97bb04098a9d7d4c3b2e9285a"
format_stamp: "Formatted at 2026-06-19 04:45:38 on dist-test-slave-2qp8"
server_key: "d19077f0acf1b0890cd56656844127c9"
server_key_iv: "8199a77c426ec5dd7bd95e4c459bc6a4"
server_key_version: "encryptionkey@0"
I20260619 04:45:38.422987 6992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 04:45:38.426352 7007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:38.427619 6992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:45:38.427807 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "1a606aa97bb04098a9d7d4c3b2e9285a"
format_stamp: "Formatted at 2026-06-19 04:45:38 on dist-test-slave-2qp8"
server_key: "d19077f0acf1b0890cd56656844127c9"
server_key_iv: "8199a77c426ec5dd7bd95e4c459bc6a4"
server_key_version: "encryptionkey@0"
I20260619 04:45:38.428083 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:38.466101 6992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:38.470260 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:38.470587 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:38.489291 6992 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:38155
I20260619 04:45:38.489326 7058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:38155 every 8 connection(s)
I20260619 04:45:38.490645 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:38.494153 7059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:38.494542 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 6992
I20260619 04:45:38.494735 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:38.495386 4046 external_mini_cluster.cc:1428] Setting key fbba5dda86db9aa326ff4c7cae6b0de3
I20260619 04:45:38.508448 7059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a: Bootstrap starting.
Jun 19 04:45:38 dist-test-slave-2qp8 krb5kdc[6979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:38.511657 7059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:38.513005 7059 log.cc:824] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:38.516248 7059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a: No bootstrap required, opened a new log
I20260619 04:45:38.519883 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:38.497375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.3.243.190:38155)
0619 04:45:38.498227 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:38.498241 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:38.498294 (+ 53us) server_negotiation.cc:407] Connection header received
0619 04:45:38.500838 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:38.500866 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:38.501387 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:38.501837 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:38.503145 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:38.505234 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:38.506482 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:38.507495 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:38.507674 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:38.511737 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:38.511748 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:38.511766 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:38.511814 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:38.515513 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:38.516164 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:38.516173 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:38.516180 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:38.516385 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:38.516755 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:38.516760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:38.516762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:38.517453 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:38.517727 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:38.518119 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 04:45:38.518550 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":375,"threads_started":1}
I20260619 04:45:38.523713 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:45:38.525568 7059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a606aa97bb04098a9d7d4c3b2e9285a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38155 } }
I20260619 04:45:38.525899 7059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:38.525995 7059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a606aa97bb04098a9d7d4c3b2e9285a, State: Initialized, Role: FOLLOWER
I20260619 04:45:38.526614 7059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a606aa97bb04098a9d7d4c3b2e9285a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38155 } }
I20260619 04:45:38.526829 7059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:38.526954 7059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:38.527122 7059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:38.529302 7059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a606aa97bb04098a9d7d4c3b2e9285a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38155 } }
I20260619 04:45:38.529779 7059 leader_election.cc:302] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a606aa97bb04098a9d7d4c3b2e9285a; no voters:
I20260619 04:45:38.530427 7059 leader_election.cc:288] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:38.530546 7065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:38.531847 7065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [term 1 LEADER]: Becoming Leader. State: Replica: 1a606aa97bb04098a9d7d4c3b2e9285a, State: Running, Role: LEADER
I20260619 04:45:38.532405 7065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a606aa97bb04098a9d7d4c3b2e9285a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38155 } }
I20260619 04:45:38.532879 7059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:38.535410 7067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a606aa97bb04098a9d7d4c3b2e9285a. Latest consensus state: current_term: 1 leader_uuid: "1a606aa97bb04098a9d7d4c3b2e9285a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a606aa97bb04098a9d7d4c3b2e9285a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38155 } } }
I20260619 04:45:38.535825 7067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:38.535562 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a606aa97bb04098a9d7d4c3b2e9285a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a606aa97bb04098a9d7d4c3b2e9285a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38155 } } }
I20260619 04:45:38.535955 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:38.548341 7073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:38.553814 7073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:38.564255 7073 catalog_manager.cc:1347] Generated new cluster ID: 25b7842b53b64284b03e771f4929b62a
I20260619 04:45:38.564373 7073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:38.571789 7062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:38.562625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54922 (local address 127.3.243.190:38155)
0619 04:45:38.562808 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:38.562816 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:38.563378 (+ 562us) server_negotiation.cc:407] Connection header received
0619 04:45:38.563600 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:38.563606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:38.563683 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:38.563799 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:38.565291 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:38.566473 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:38.568281 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:38.568950 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:38.569099 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:38.570214 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:38.570222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:38.570227 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:38.570990 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:38.571073 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:38.571108 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:38.571222 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:38.571349 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 04:45:38.571450 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:45:38.591888 7073 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:38.593560 7073 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:38.600051 7073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a606aa97bb04098a9d7d4c3b2e9285a: Generated new TSK 0
I20260619 04:45:38.601089 7073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:45:38.613562 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 6992
I20260619 04:45:38.614864 7054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:38.705102 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 6992
2026-06-19T04:45:38Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1934 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:38 dist-test-slave-2qp8 krb5kdc[7093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:38 dist-test-slave-2qp8 krb5kdc[7093](info): set up 2 sockets
Jun 19 04:45:38 dist-test-slave-2qp8 krb5kdc[7093](info): commencing operation
krb5kdc: starting...
W20260619 04:45:39.780462 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:39 dist-test-slave-2qp8 krb5kdc[7093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:39.975641 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:39427
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:39427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:40.294399 7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:40.294747 7106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:40.294982 7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:40.303103 7106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:40.303223 7106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:40.303329 7106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:40.303464 7106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:40.303529 7106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:40.313679 7106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:39427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:39427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:40.315135 7106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:40.316514 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:40.329597 7111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:40.329574 7114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:40.329643 7112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:40.330394 7106 server_base.cc:1034] running on GCE node
I20260619 04:45:40.331110 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:40.332577 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:40.333808 7106 hybrid_clock.cc:648] HybridClock initialized: now 1781844340333766 us; error 54 us; skew 500 ppm
I20260619 04:45:40.334160 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:40 dist-test-slave-2qp8 krb5kdc[7093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:40.337314 7106 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:40.338693 7106 webserver.cc:460] Webserver started at http://127.3.243.190:38959/ using document root <none> and password file <none>
I20260619 04:45:40.339406 7106 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:40.339550 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:40.339854 7106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:40.342932 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "ae50193adc1e40ab8a0e442f21472040"
format_stamp: "Formatted at 2026-06-19 04:45:40 on dist-test-slave-2qp8"
server_key: "edd07ce8b47d65ffe300c6477f8a741f"
server_key_iv: "03c360a8419d6ce3cb2622420d6d3020"
server_key_version: "encryptionkey@0"
I20260619 04:45:40.343676 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "ae50193adc1e40ab8a0e442f21472040"
format_stamp: "Formatted at 2026-06-19 04:45:40 on dist-test-slave-2qp8"
server_key: "edd07ce8b47d65ffe300c6477f8a741f"
server_key_iv: "03c360a8419d6ce3cb2622420d6d3020"
server_key_version: "encryptionkey@0"
I20260619 04:45:40.348995 7106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:45:40.352517 7121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:40.353790 7106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:45:40.353982 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "ae50193adc1e40ab8a0e442f21472040"
format_stamp: "Formatted at 2026-06-19 04:45:40 on dist-test-slave-2qp8"
server_key: "edd07ce8b47d65ffe300c6477f8a741f"
server_key_iv: "03c360a8419d6ce3cb2622420d6d3020"
server_key_version: "encryptionkey@0"
I20260619 04:45:40.354189 7106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:40.366597 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:40.370880 7106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:40.371170 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:40.389870 7106 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:39427
I20260619 04:45:40.389912 7172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:39427 every 8 connection(s)
I20260619 04:45:40.391263 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:40.392890 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 7106
I20260619 04:45:40.393128 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:40.393776 4046 external_mini_cluster.cc:1428] Setting key c7fa56c29e574fd5c92aec6d55a05e35
I20260619 04:45:40.395539 7173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:45:40 dist-test-slave-2qp8 krb5kdc[7093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:40.408699 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040: Bootstrap starting.
I20260619 04:45:40.412748 7173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:40.413997 7173 log.cc:824] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:40.417193 7173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040: No bootstrap required, opened a new log
I20260619 04:45:40.417970 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:40.395788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.3.243.190:39427)
0619 04:45:40.396565 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:40.396580 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:40.396627 (+ 47us) server_negotiation.cc:407] Connection header received
0619 04:45:40.398642 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:40.398669 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:40.399222 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:40.399647 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:40.400947 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:40.402938 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:40.404167 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:40.405293 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:40.405475 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:40.409399 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:40.409409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:40.409427 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:40.409473 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:40.413358 (+ 3885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:40.413998 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:40.414005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:40.414015 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:40.414219 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:40.414764 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:40.414771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:40.414775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:40.415635 (+ 860us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:40.415953 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:40.416484 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 04:45:40.416906 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":321,"threads_started":1}
I20260619 04:45:40.424834 7173 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50193adc1e40ab8a0e442f21472040" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39427 } }
I20260619 04:45:40.425203 7173 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:40.425310 7173 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae50193adc1e40ab8a0e442f21472040, State: Initialized, Role: FOLLOWER
I20260619 04:45:40.425877 7173 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50193adc1e40ab8a0e442f21472040" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39427 } }
I20260619 04:45:40.426116 7173 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:40.426254 7173 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:40.426422 7173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:40.428503 7173 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50193adc1e40ab8a0e442f21472040" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39427 } }
I20260619 04:45:40.428922 7173 leader_election.cc:302] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae50193adc1e40ab8a0e442f21472040; no voters:
I20260619 04:45:40.429477 7173 leader_election.cc:288] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:40.429595 7178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:40.430315 7178 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [term 1 LEADER]: Becoming Leader. State: Replica: ae50193adc1e40ab8a0e442f21472040, State: Running, Role: LEADER
I20260619 04:45:40.430879 7178 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50193adc1e40ab8a0e442f21472040" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39427 } }
I20260619 04:45:40.431640 7173 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:40.434336 7180 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae50193adc1e40ab8a0e442f21472040. Latest consensus state: current_term: 1 leader_uuid: "ae50193adc1e40ab8a0e442f21472040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50193adc1e40ab8a0e442f21472040" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39427 } } }
I20260619 04:45:40.434365 7179 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae50193adc1e40ab8a0e442f21472040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50193adc1e40ab8a0e442f21472040" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39427 } } }
I20260619 04:45:40.434744 7180 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:40.434756 7179 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:40.435591 7187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:40.440805 7187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:40.449947 7187 catalog_manager.cc:1347] Generated new cluster ID: ab29777cb8d449f493af164391f64878
I20260619 04:45:40.450094 7187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:40.475052 7187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:40.476486 7187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:40.497583 7187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae50193adc1e40ab8a0e442f21472040: Generated new TSK 0
I20260619 04:45:40.498448 7187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:45:40.543769 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:39427
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:40.872736 7199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:40.873021 7199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:40.873113 7199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:40.880609 7199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:40.880717 7199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:40.880883 7199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:40.891319 7199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:39427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:40.892627 7199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:40.893882 7199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:40.907235 7204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:40.907311 7205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:40.907334 7207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:40.908054 7199 server_base.cc:1034] running on GCE node
I20260619 04:45:40.908670 7199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:40.909582 7199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:40.910775 7199 hybrid_clock.cc:648] HybridClock initialized: now 1781844340910709 us; error 72 us; skew 500 ppm
I20260619 04:45:40.911077 7199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:40 dist-test-slave-2qp8 krb5kdc[7093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:40.914566 7199 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:45:40.916041 7199 webserver.cc:460] Webserver started at http://127.3.243.129:44219/ using document root <none> and password file <none>
I20260619 04:45:40.916769 7199 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:40.916910 7199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:40.917259 7199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:40.920022 7199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "134e260adc01489cb8e8479b2690e63d"
format_stamp: "Formatted at 2026-06-19 04:45:40 on dist-test-slave-2qp8"
server_key: "5532bae8eaba107a63e9fbac268a3e5a"
server_key_iv: "e497ef7a6d44103e3224efd29d5d50b2"
server_key_version: "encryptionkey@0"
I20260619 04:45:40.920826 7199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "134e260adc01489cb8e8479b2690e63d"
format_stamp: "Formatted at 2026-06-19 04:45:40 on dist-test-slave-2qp8"
server_key: "5532bae8eaba107a63e9fbac268a3e5a"
server_key_iv: "e497ef7a6d44103e3224efd29d5d50b2"
server_key_version: "encryptionkey@0"
I20260619 04:45:40.925890 7199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:45:40.929214 7214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:40.930464 7199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:45:40.930640 7199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "134e260adc01489cb8e8479b2690e63d"
format_stamp: "Formatted at 2026-06-19 04:45:40 on dist-test-slave-2qp8"
server_key: "5532bae8eaba107a63e9fbac268a3e5a"
server_key_iv: "e497ef7a6d44103e3224efd29d5d50b2"
server_key_version: "encryptionkey@0"
I20260619 04:45:40.930823 7199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:40.945442 7199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:40.949133 7199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:40.949420 7199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:40.950292 7199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:40.951555 7199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:40.951651 7199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:40.951795 7199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:40.951867 7199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:40.984424 7199 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:41249
I20260619 04:45:40.984462 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:41249 every 8 connection(s)
I20260619 04:45:40.985711 7199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:40.991910 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 7199
I20260619 04:45:40.992174 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:40.992796 4046 external_mini_cluster.cc:1428] Setting key 7f1890c2c0903a5049c3d1860ca01470
Jun 19 04:45:40 dist-test-slave-2qp8 krb5kdc[7093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:41.005910 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:40.988233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:50497 (local address 127.3.243.190:39427)
0619 04:45:40.988676 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:40.988681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:40.989506 (+ 825us) server_negotiation.cc:407] Connection header received
0619 04:45:40.991290 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:40.991296 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:40.991382 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:40.991502 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:40.993219 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:40.994103 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:40.995784 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:40.996326 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:40.996457 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:41.000203 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:41.000208 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:41.000212 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:41.000241 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:41.002648 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:41.003606 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:41.003610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:41.003613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:41.003719 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:41.004209 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:41.004213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:41.004215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:41.004483 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:41.004605 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:41.005485 (+ 880us) server_negotiation.cc:299] Negotiation successful
0619 04:45:41.005653 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":256,"threads_started":1}
I20260619 04:45:41.007609 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:40.988661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39427 (local address 127.3.243.129:50497)
0619 04:45:40.989347 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:40.989385 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:40.991014 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:40.991760 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:40.991769 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:40.992400 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:40.993058 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:40.993077 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:40.994250 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:40.994258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:40.995630 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:40.995641 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:40.996553 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:40.996563 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:40.996804 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:40.997489 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:40.997511 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:40.999925 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:41.002799 (+ 2874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:41.002812 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:41.002833 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:41.003485 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:41.003821 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:41.003828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:41.003835 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:41.004086 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:41.004613 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:41.004624 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:41.005092 (+ 468us) client_negotiation.cc:769] Sending connection context
0619 04:45:41.005392 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 04:45:41.005976 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":267,"threads_started":1}
I20260619 04:45:41.009298 7327 heartbeater.cc:347] Connected to a master server at 127.3.243.190:39427
I20260619 04:45:41.009613 7327 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:41.010337 7327 heartbeater.cc:511] Master 127.3.243.190:39427 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:45:41.012460 7138 ts_manager.cc:194] Registered new tserver with Master: 134e260adc01489cb8e8479b2690e63d (127.3.243.129:41249)
I20260619 04:45:41.013860 7138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:50497
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:45:41.036777 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:39427
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:41.353407 7333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:41.353722 7333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:41.353899 7333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:41.361531 7333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:41.361686 7333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:41.361892 7333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:41.372494 7333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:39427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:41.374133 7333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:41.375864 7333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:41.390249 7339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:41.390277 7338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:41.391640 7333 server_base.cc:1034] running on GCE node
W20260619 04:45:41.396764 7341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:41.397511 7333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:41.398633 7333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:41.399854 7333 hybrid_clock.cc:648] HybridClock initialized: now 1781844341399811 us; error 41 us; skew 500 ppm
I20260619 04:45:41.400243 7333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:41 dist-test-slave-2qp8 krb5kdc[7093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844341, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:41.403404 7333 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:45:41.405071 7333 webserver.cc:460] Webserver started at http://127.3.243.130:40729/ using document root <none> and password file <none>
I20260619 04:45:41.405791 7333 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:41.405925 7333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:41.406224 7333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:41.409116 7333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "7c0b909897f64da8a52dfe9094d26d73"
format_stamp: "Formatted at 2026-06-19 04:45:41 on dist-test-slave-2qp8"
server_key: "391c230cabc2f7dc7c424ebb2cf6ddb1"
server_key_iv: "4d1d4c31883b8bb15efa5af4969e9072"
server_key_version: "encryptionkey@0"
I20260619 04:45:41.409873 7333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "7c0b909897f64da8a52dfe9094d26d73"
format_stamp: "Formatted at 2026-06-19 04:45:41 on dist-test-slave-2qp8"
server_key: "391c230cabc2f7dc7c424ebb2cf6ddb1"
server_key_iv: "4d1d4c31883b8bb15efa5af4969e9072"
server_key_version: "encryptionkey@0"
I20260619 04:45:41.414830 7333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:45:41.418123 7348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:41.419454 7333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:41.419622 7333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "7c0b909897f64da8a52dfe9094d26d73"
format_stamp: "Formatted at 2026-06-19 04:45:41 on dist-test-slave-2qp8"
server_key: "391c230cabc2f7dc7c424ebb2cf6ddb1"
server_key_iv: "4d1d4c31883b8bb15efa5af4969e9072"
server_key_version: "encryptionkey@0"
I20260619 04:45:41.419813 7333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:41.436758 7333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:41.441128 7333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:41.441424 7333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:41.442341 7333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:41.443682 7333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:41.443786 7333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:41.443939 7333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:41.444011 7333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:41.477206 7333 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:33045
I20260619 04:45:41.477244 7460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:33045 every 8 connection(s)
I20260619 04:45:41.478503 7333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:41.484015 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 7333
I20260619 04:45:41.484256 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:41.484802 4046 external_mini_cluster.cc:1428] Setting key 1336092681e8ddf65668649106dcf79b
Jun 19 04:45:41 dist-test-slave-2qp8 krb5kdc[7093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844341, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:41.498632 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:41.481197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:35591 (local address 127.3.243.190:39427)
0619 04:45:41.481407 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:41.481413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:41.482511 (+ 1098us) server_negotiation.cc:407] Connection header received
0619 04:45:41.484310 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:41.484316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:41.484406 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:41.484522 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:41.486153 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:41.487197 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:41.488792 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:41.489515 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:41.489658 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:41.493087 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:41.493091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:41.493095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:41.493124 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:41.495646 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:41.496403 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:41.496408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:41.496411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:41.496548 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:41.496979 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:41.496983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:41.496985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:41.497260 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:41.497421 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:41.498076 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 04:45:41.498294 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":71}
I20260619 04:45:41.499527 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:41.481654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39427 (local address 127.3.243.130:35591)
0619 04:45:41.482345 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:41.482387 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:41.483979 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:41.484706 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:41.484714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:41.485305 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:41.485998 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:41.486017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:41.487356 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:41.487360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:41.488656 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:41.488668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:41.489717 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:41.489721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:41.489914 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:41.490566 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:41.490592 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:41.492858 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:41.495784 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:41.495793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:41.495809 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:41.496282 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:41.496666 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:41.496671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:41.496675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:41.496851 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:41.497387 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:41.497395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:41.497723 (+ 328us) client_negotiation.cc:769] Sending connection context
0619 04:45:41.497974 (+ 251us) client_negotiation.cc:240] Negotiation successful
0619 04:45:41.498481 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":277,"threads_started":1}
I20260619 04:45:41.501077 7461 heartbeater.cc:347] Connected to a master server at 127.3.243.190:39427
I20260619 04:45:41.501407 7461 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:41.502318 7461 heartbeater.cc:511] Master 127.3.243.190:39427 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:45:41.503952 7138 ts_manager.cc:194] Registered new tserver with Master: 7c0b909897f64da8a52dfe9094d26d73 (127.3.243.130:33045)
I20260619 04:45:41.504829 7138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:35591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:45:41.528198 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:39427
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:41.847822 7466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:41.848099 7466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:41.848335 7466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:41.855415 7466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:41.855516 7466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:41.855662 7466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:45:41.865406 7466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:39427
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:41.866734 7466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:41.868083 7466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:41.881366 7472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:41.881448 7474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:41.881470 7471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:41.885684 7466 server_base.cc:1034] running on GCE node
I20260619 04:45:41.886312 7466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:41.887241 7466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:41.888437 7466 hybrid_clock.cc:648] HybridClock initialized: now 1781844341888393 us; error 29 us; skew 500 ppm
I20260619 04:45:41.888768 7466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:41 dist-test-slave-2qp8 krb5kdc[7093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844341, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:41.891944 7466 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:45:41.893504 7466 webserver.cc:460] Webserver started at http://127.3.243.131:36563/ using document root <none> and password file <none>
I20260619 04:45:41.894208 7466 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:41.894335 7466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:41.894640 7466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:41.897562 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "2cae32e313c3471f9a783efd18df6aef"
format_stamp: "Formatted at 2026-06-19 04:45:41 on dist-test-slave-2qp8"
server_key: "1f95bfadff5b0200838b07eaa19c2e80"
server_key_iv: "33d59f02de2714d3e38dd2ecfef903f3"
server_key_version: "encryptionkey@0"
I20260619 04:45:41.898340 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "2cae32e313c3471f9a783efd18df6aef"
format_stamp: "Formatted at 2026-06-19 04:45:41 on dist-test-slave-2qp8"
server_key: "1f95bfadff5b0200838b07eaa19c2e80"
server_key_iv: "33d59f02de2714d3e38dd2ecfef903f3"
server_key_version: "encryptionkey@0"
I20260619 04:45:41.904021 7466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 04:45:41.907945 7481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:41.909215 7466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:41.909373 7466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "2cae32e313c3471f9a783efd18df6aef"
format_stamp: "Formatted at 2026-06-19 04:45:41 on dist-test-slave-2qp8"
server_key: "1f95bfadff5b0200838b07eaa19c2e80"
server_key_iv: "33d59f02de2714d3e38dd2ecfef903f3"
server_key_version: "encryptionkey@0"
I20260619 04:45:41.909580 7466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:41.930177 7466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:41.934809 7466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:41.935143 7466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:41.936319 7466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:41.937649 7466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:41.937752 7466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:41.937901 7466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:41.937970 7466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:41.970424 7466 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:40275
I20260619 04:45:41.970489 7593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:40275 every 8 connection(s)
I20260619 04:45:41.971758 7466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:45:41.976199 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 7466
I20260619 04:45:41.976372 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:45:41.976826 4046 external_mini_cluster.cc:1428] Setting key 35bf9587d571282aa9a12dc08bb604aa
Jun 19 04:45:41 dist-test-slave-2qp8 krb5kdc[7093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844341, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:41.991292 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:41.974236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:47651 (local address 127.3.243.190:39427)
0619 04:45:41.974438 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:41.974443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:41.975537 (+ 1094us) server_negotiation.cc:407] Connection header received
0619 04:45:41.977487 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:41.977491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:41.977562 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:41.977642 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:41.979210 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:41.980156 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:41.981740 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:41.982145 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:41.982235 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:41.985464 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:41.985469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:41.985472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:41.985504 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:41.988248 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:41.988947 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:41.988953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:41.988957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:41.989085 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:41.989494 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:41.989501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:41.989506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:41.989820 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:41.989981 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:41.990695 (+ 714us) server_negotiation.cc:299] Negotiation successful
0619 04:45:41.990927 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260619 04:45:41.992273 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:41.974705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39427 (local address 127.3.243.131:47651)
0619 04:45:41.975387 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:41.975424 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:41.977232 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:41.977900 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:41.977912 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:41.978444 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:41.979068 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:41.979084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:41.980294 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:41.980299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:41.981623 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:41.981634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:41.982313 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:41.982317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:41.982510 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:41.983167 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:41.983188 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:41.985255 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:41.988389 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:41.988398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:41.988412 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:41.988822 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:41.989195 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:41.989201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:41.989205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:41.989384 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:41.989953 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:41.989959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:41.990317 (+ 358us) client_negotiation.cc:769] Sending connection context
0619 04:45:41.990616 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 04:45:41.991116 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":269,"threads_started":1}
I20260619 04:45:41.993543 7594 heartbeater.cc:347] Connected to a master server at 127.3.243.190:39427
I20260619 04:45:41.993853 7594 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:41.994565 7594 heartbeater.cc:511] Master 127.3.243.190:39427 requested a full tablet report, sending...
I20260619 04:45:41.995868 7138 ts_manager.cc:194] Registered new tserver with Master: 2cae32e313c3471f9a783efd18df6aef (127.3.243.131:40275)
I20260619 04:45:41.996742 7138 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:47651
I20260619 04:45:42.004243 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:45:42.016883 7327 heartbeater.cc:503] Master 127.3.243.190:39427 was elected leader, sending a full tablet report...
Jun 19 04:45:42 dist-test-slave-2qp8 krb5kdc[7093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844342, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 04:45:42 dist-test-slave-2qp8 krb5kdc[7093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844342, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:42.053349 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:42.039557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.3.243.190:39427)
0619 04:45:42.039829 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:42.039839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:42.040258 (+ 419us) server_negotiation.cc:407] Connection header received
0619 04:45:42.040442 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:42.040446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:42.040515 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:42.040603 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:42.041893 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:42.042636 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:42.043862 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:42.044328 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:42.044423 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:42.048241 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:42.048246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:42.048250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:42.048282 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:42.050901 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:42.051476 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:42.051480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:42.051482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:42.051629 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:42.051971 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:42.051975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:42.051977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:42.052378 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:42.052567 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:42.052842 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 04:45:42.053012 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 04:45:42.053941 7138 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59432
I20260619 04:45:42.055771 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7199
Jun 19 04:45:42 dist-test-slave-2qp8 krb5kdc[7093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844342, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:45:42.071012 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:42.057051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.3.243.129:41249)
0619 04:45:42.057707 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:42.057714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:42.057738 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:45:42.057792 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:42.057798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:42.058011 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:42.058134 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:42.059275 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:42.060387 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:42.061544 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:42.062115 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:42.062211 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:42.066009 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:42.066022 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:42.066036 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:42.066074 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:42.068689 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:42.069251 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:42.069257 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:42.069259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:42.069368 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:42.069666 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:42.069669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:42.069671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:42.069980 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:42.070192 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:42.070416 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 04:45:42.070619 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":340,"threads_started":1}
W20260619 04:45:42.072350 7322 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58322
W20260619 04:45:42.073450 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7199: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:45:42.073670 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 7199
I20260619 04:45:42.089383 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7333
Jun 19 04:45:42 dist-test-slave-2qp8 krb5kdc[7093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844342, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:45:42.104368 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:42.090288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.3.243.130:33045)
0619 04:45:42.090759 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:42.090766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:42.090790 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:45:42.090849 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:42.090855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:42.091074 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:42.091196 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:42.092439 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:42.093426 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:42.094582 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:42.095119 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:42.095215 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:42.099166 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:42.099176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:42.099190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:42.099221 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:42.102059 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:42.102607 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:42.102613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:42.102615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:42.102725 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:42.103023 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:42.103026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:42.103028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:42.103332 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:42.103550 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:42.103748 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 04:45:42.103927 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":235,"threads_started":1}
W20260619 04:45:42.105585 7456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42742
W20260619 04:45:42.106681 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7333: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:45:42.106845 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 7333
I20260619 04:45:42.122097 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7466
Jun 19 04:45:42 dist-test-slave-2qp8 krb5kdc[7093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844342, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:45:42.138309 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:42.123223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.3.243.131:40275)
0619 04:45:42.123898 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:42.123905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:42.123933 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:45:42.124007 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:42.124012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:42.124318 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:42.124449 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:42.125710 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:42.126737 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:42.127953 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:42.128577 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:42.128673 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:42.132891 (+ 4218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:42.132902 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:42.132916 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:42.132951 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:42.135855 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:42.136496 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:42.136502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:42.136505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:42.136619 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:42.136904 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:42.136907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:42.136908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:42.137228 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:42.137441 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:42.137669 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 04:45:42.137870 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":379,"threads_started":1}
W20260619 04:45:42.139604 7589 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56158
W20260619 04:45:42.140892 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7466: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:45:42.141054 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 7466
I20260619 04:45:42.156215 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7106
I20260619 04:45:42.157331 7168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:42.263643 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 7106
2026-06-19T04:45:42Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3564 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:42 dist-test-slave-2qp8 krb5kdc[7611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:42 dist-test-slave-2qp8 krb5kdc[7611](info): set up 2 sockets
Jun 19 04:45:42 dist-test-slave-2qp8 krb5kdc[7611](info): commencing operation
krb5kdc: starting...
W20260619 04:45:43.440646 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.003s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:43 dist-test-slave-2qp8 krb5kdc[7611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:43.627135 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:43333
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:43333
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:43.936852 7624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:43.937129 7624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:43.937217 7624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:43.944294 7624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:43.944397 7624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:43.944450 7624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:43.944502 7624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:43.944548 7624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:43.954077 7624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:43333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:43333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:43.955302 7624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:43.956533 7624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:43.968140 7630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:43.968189 7629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:43.968077 7632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:43.973050 7624 server_base.cc:1034] running on GCE node
I20260619 04:45:43.973853 7624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:43.975317 7624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:43.976540 7624 hybrid_clock.cc:648] HybridClock initialized: now 1781844343976491 us; error 59 us; skew 500 ppm
I20260619 04:45:43.976881 7624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:43 dist-test-slave-2qp8 krb5kdc[7611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844343, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:43.984524 7624 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:43.986073 7624 webserver.cc:460] Webserver started at http://127.3.243.190:42775/ using document root <none> and password file <none>
I20260619 04:45:43.986757 7624 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:43.986853 7624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:43.987110 7624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:43.990092 7624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "997a56fcce12461caa7c435dee1e79cb"
format_stamp: "Formatted at 2026-06-19 04:45:43 on dist-test-slave-2qp8"
server_key: "11e06a43e4404a2889061d73b3b1e8ea"
server_key_iv: "f7bad999e4b7f5ceb68f24b2555a341f"
server_key_version: "encryptionkey@0"
I20260619 04:45:43.990819 7624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "997a56fcce12461caa7c435dee1e79cb"
format_stamp: "Formatted at 2026-06-19 04:45:43 on dist-test-slave-2qp8"
server_key: "11e06a43e4404a2889061d73b3b1e8ea"
server_key_iv: "f7bad999e4b7f5ceb68f24b2555a341f"
server_key_version: "encryptionkey@0"
I20260619 04:45:43.995859 7624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 04:45:43.999173 7639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:44.000584 7624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:44.000766 7624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "997a56fcce12461caa7c435dee1e79cb"
format_stamp: "Formatted at 2026-06-19 04:45:43 on dist-test-slave-2qp8"
server_key: "11e06a43e4404a2889061d73b3b1e8ea"
server_key_iv: "f7bad999e4b7f5ceb68f24b2555a341f"
server_key_version: "encryptionkey@0"
I20260619 04:45:44.000962 7624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:44.028841 7624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:44.033327 7624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:44.033646 7624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:44.051563 7624 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:43333
I20260619 04:45:44.051585 7690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:43333 every 8 connection(s)
I20260619 04:45:44.052929 7624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:44.053490 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 7624
I20260619 04:45:44.053759 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:44.054498 4046 external_mini_cluster.cc:1428] Setting key 3bca4069ce6a6002a32c3759999bc2c0
I20260619 04:45:44.057224 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:45:44 dist-test-slave-2qp8 krb5kdc[7611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:44.070024 7691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb: Bootstrap starting.
I20260619 04:45:44.073084 7691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:44.074079 7691 log.cc:824] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:44.076862 7691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb: No bootstrap required, opened a new log
I20260619 04:45:44.076987 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:44.056545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.3.243.190:43333)
0619 04:45:44.057381 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:44.057394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:44.057435 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:45:44.059031 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:44.059051 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:44.059416 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:44.059759 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:44.061586 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:44.062906 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:44.064747 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:44.065377 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:44.065499 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:44.070068 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:44.070075 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:44.070088 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:44.070157 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:44.073117 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:44.073794 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:44.073801 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:44.073807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:44.073972 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:44.074382 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:44.074386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:44.074388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:44.074865 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:44.075194 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:44.075596 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 04:45:44.075975 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":349,"threads_started":1}
I20260619 04:45:44.084853 7691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997a56fcce12461caa7c435dee1e79cb" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43333 } }
I20260619 04:45:44.085220 7691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:44.085395 7691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 997a56fcce12461caa7c435dee1e79cb, State: Initialized, Role: FOLLOWER
I20260619 04:45:44.086035 7691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997a56fcce12461caa7c435dee1e79cb" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43333 } }
I20260619 04:45:44.086265 7691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:44.086390 7691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:44.086566 7691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:44.088708 7691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997a56fcce12461caa7c435dee1e79cb" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43333 } }
I20260619 04:45:44.089200 7691 leader_election.cc:302] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 997a56fcce12461caa7c435dee1e79cb; no voters:
I20260619 04:45:44.089733 7691 leader_election.cc:288] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:44.089861 7696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:44.090530 7696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [term 1 LEADER]: Becoming Leader. State: Replica: 997a56fcce12461caa7c435dee1e79cb, State: Running, Role: LEADER
I20260619 04:45:44.091025 7696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997a56fcce12461caa7c435dee1e79cb" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43333 } }
I20260619 04:45:44.091756 7691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:44.094101 7697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "997a56fcce12461caa7c435dee1e79cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997a56fcce12461caa7c435dee1e79cb" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43333 } } }
I20260619 04:45:44.094475 7697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:44.094882 7698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 997a56fcce12461caa7c435dee1e79cb. Latest consensus state: current_term: 1 leader_uuid: "997a56fcce12461caa7c435dee1e79cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "997a56fcce12461caa7c435dee1e79cb" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43333 } } }
I20260619 04:45:44.095180 7698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:44.095290 7702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:44.100847 7702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:44.109942 7702 catalog_manager.cc:1347] Generated new cluster ID: 51d72be83bb94ff0b8f7453055d11872
I20260619 04:45:44.110050 7702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:44.154101 7702 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:44.156001 7702 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:44.168138 7702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 997a56fcce12461caa7c435dee1e79cb: Generated new TSK 0
I20260619 04:45:44.169008 7702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:45:44.236066 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43333
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:44.559407 7717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:44.559729 7717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:44.559890 7717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:44.567186 7717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:44.567339 7717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:44.567510 7717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:44.577370 7717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:44.578593 7717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:44.579896 7717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:44.593240 7723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:44.593261 7725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:44.593567 7722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:44.594208 7717 server_base.cc:1034] running on GCE node
I20260619 04:45:44.594830 7717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:44.595791 7717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:44.597038 7717 hybrid_clock.cc:648] HybridClock initialized: now 1781844344596993 us; error 37 us; skew 500 ppm
I20260619 04:45:44.597462 7717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:44 dist-test-slave-2qp8 krb5kdc[7611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:44.600593 7717 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:45:44.601938 7717 webserver.cc:460] Webserver started at http://127.3.243.129:36327/ using document root <none> and password file <none>
I20260619 04:45:44.602573 7717 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:44.602669 7717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:44.602936 7717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:44.605902 7717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "cb14381a7cb94a03a54664880adb6c88"
format_stamp: "Formatted at 2026-06-19 04:45:44 on dist-test-slave-2qp8"
server_key: "d320c4de460979a42fbc15433b9b1b12"
server_key_iv: "e592226bb4f6ea6fd8433492401b08df"
server_key_version: "encryptionkey@0"
I20260619 04:45:44.606599 7717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "cb14381a7cb94a03a54664880adb6c88"
format_stamp: "Formatted at 2026-06-19 04:45:44 on dist-test-slave-2qp8"
server_key: "d320c4de460979a42fbc15433b9b1b12"
server_key_iv: "e592226bb4f6ea6fd8433492401b08df"
server_key_version: "encryptionkey@0"
I20260619 04:45:44.611565 7717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:45:44.614953 7732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:44.616441 7717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 04:45:44.616577 7717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "cb14381a7cb94a03a54664880adb6c88"
format_stamp: "Formatted at 2026-06-19 04:45:44 on dist-test-slave-2qp8"
server_key: "d320c4de460979a42fbc15433b9b1b12"
server_key_iv: "e592226bb4f6ea6fd8433492401b08df"
server_key_version: "encryptionkey@0"
I20260619 04:45:44.616844 7717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:44.631525 7717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:44.635370 7717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:44.635656 7717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:44.636564 7717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:44.637799 7717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:44.637900 7717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:44.638063 7717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:44.638137 7717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:44.670612 7717 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:32885
I20260619 04:45:44.670656 7844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:32885 every 8 connection(s)
I20260619 04:45:44.672035 7717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:44.676162 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 7717
I20260619 04:45:44.676383 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:44.677016 4046 external_mini_cluster.cc:1428] Setting key f90aeef46c23538e05963f6911b13138
Jun 19 04:45:44 dist-test-slave-2qp8 krb5kdc[7611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844344, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:44.693601 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:44.674618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:47371 (local address 127.3.243.190:43333)
0619 04:45:44.675047 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:44.675051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:44.676008 (+ 957us) server_negotiation.cc:407] Connection header received
0619 04:45:44.678102 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:44.678106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:44.678168 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:44.678257 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:44.679941 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:44.680740 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:44.682250 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:44.682670 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:44.682804 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:44.686507 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:44.686515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:44.686522 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:44.686566 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:44.690074 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:44.690790 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:44.690798 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:44.690804 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:44.690955 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:44.691358 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:44.691365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:44.691372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:44.691743 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:44.691940 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:44.692785 (+ 845us) server_negotiation.cc:299] Negotiation successful
0619 04:45:44.693097 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":231,"threads_started":1}
I20260619 04:45:44.694248 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:44.675129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:43333 (local address 127.3.243.129:47371)
0619 04:45:44.675832 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:44.675872 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:44.677833 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:44.678536 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:44.678547 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:44.679222 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:44.679777 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:44.679794 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:44.680906 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:44.680910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:44.682108 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:44.682119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:44.682894 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:44.682902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:44.683136 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:44.683847 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:44.683869 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:44.686294 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:44.690230 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:44.690236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:44.690249 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:44.690671 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:44.691073 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:44.691076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:44.691078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:44.691246 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:44.691873 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:44.691879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:44.692388 (+ 509us) client_negotiation.cc:769] Sending connection context
0619 04:45:44.692632 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 04:45:44.693129 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":272,"threads_started":1}
I20260619 04:45:44.695851 7845 heartbeater.cc:347] Connected to a master server at 127.3.243.190:43333
I20260619 04:45:44.696233 7845 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:45:44.697100 7845 heartbeater.cc:511] Master 127.3.243.190:43333 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
I20260619 04:45:44.699437 7656 ts_manager.cc:194] Registered new tserver with Master: cb14381a7cb94a03a54664880adb6c88 (127.3.243.129:32885)
I20260619 04:45:44.701622 7656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:47371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:45:44.722889 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43333
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:45.038734 7851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:45.039008 7851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:45.039177 7851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:45.047060 7851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:45.047191 7851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:45.047382 7851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:45.057184 7851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:45.058386 7851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:45.059666 7851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:45.072618 7856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:45.072647 7859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:45.072579 7857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:45.073659 7851 server_base.cc:1034] running on GCE node
I20260619 04:45:45.074254 7851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:45.075093 7851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:45.076311 7851 hybrid_clock.cc:648] HybridClock initialized: now 1781844345076272 us; error 54 us; skew 500 ppm
I20260619 04:45:45.076622 7851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:45 dist-test-slave-2qp8 krb5kdc[7611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:45.079694 7851 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:45:45.081104 7851 webserver.cc:460] Webserver started at http://127.3.243.130:35697/ using document root <none> and password file <none>
I20260619 04:45:45.081758 7851 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:45.081899 7851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:45.082204 7851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:45.085028 7851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "b5208d71fa9547f48092387d5a259a7a"
format_stamp: "Formatted at 2026-06-19 04:45:45 on dist-test-slave-2qp8"
server_key: "924212ec0cb0072ecfcf9768f6fbf9c1"
server_key_iv: "12ad451cf7680c136f46c08437acf2e4"
server_key_version: "encryptionkey@0"
I20260619 04:45:45.085785 7851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "b5208d71fa9547f48092387d5a259a7a"
format_stamp: "Formatted at 2026-06-19 04:45:45 on dist-test-slave-2qp8"
server_key: "924212ec0cb0072ecfcf9768f6fbf9c1"
server_key_iv: "12ad451cf7680c136f46c08437acf2e4"
server_key_version: "encryptionkey@0"
I20260619 04:45:45.091058 7851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:45:45.094548 7866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:45.095785 7851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:45.095957 7851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "b5208d71fa9547f48092387d5a259a7a"
format_stamp: "Formatted at 2026-06-19 04:45:45 on dist-test-slave-2qp8"
server_key: "924212ec0cb0072ecfcf9768f6fbf9c1"
server_key_iv: "12ad451cf7680c136f46c08437acf2e4"
server_key_version: "encryptionkey@0"
I20260619 04:45:45.096185 7851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:45.122478 7851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:45.127367 7851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:45.127674 7851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:45.128726 7851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:45.130133 7851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:45.130232 7851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:45.130388 7851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:45.130461 7851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:45.163241 7851 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:35075
I20260619 04:45:45.163285 7978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:35075 every 8 connection(s)
I20260619 04:45:45.164726 7851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:45.170970 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 7851
I20260619 04:45:45.171180 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:45.171734 4046 external_mini_cluster.cc:1428] Setting key b86838c6269a2d04e5e5bd42dcd1d3eb
Jun 19 04:45:45 dist-test-slave-2qp8 krb5kdc[7611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:45.186686 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:45.167313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:46473 (local address 127.3.243.190:43333)
0619 04:45:45.167488 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:45.167492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:45.168752 (+ 1260us) server_negotiation.cc:407] Connection header received
0619 04:45:45.170740 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:45.170745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:45.170813 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:45.170905 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:45.172873 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.173605 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.175169 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.175937 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.176079 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:45.179825 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:45.179830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:45.179833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:45.179865 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:45.183256 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.184031 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.184040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.184047 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.184255 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.184703 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.184711 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.184717 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.185108 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:45.185273 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:45.185940 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 04:45:45.186211 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:45:45.187506 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:45.167830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:43333 (local address 127.3.243.130:46473)
0619 04:45:45.168587 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:45.168624 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:45.170410 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:45.171208 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:45.171223 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:45.171970 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:45.172695 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:45.172713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.173778 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.173782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:45.175012 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:45.175023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.176211 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.176217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:45.176415 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:45.177186 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:45.177209 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:45.179598 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:45.183417 (+ 3819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:45.183425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:45.183439 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:45.183885 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:45.184376 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:45.184379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:45.184381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:45.184582 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:45.185234 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:45.185240 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:45.185569 (+ 329us) client_negotiation.cc:769] Sending connection context
0619 04:45:45.185790 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 04:45:45.186203 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":341,"threads_started":1}
I20260619 04:45:45.189059 7979 heartbeater.cc:347] Connected to a master server at 127.3.243.190:43333
I20260619 04:45:45.189404 7979 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:45.190254 7979 heartbeater.cc:511] Master 127.3.243.190:43333 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:45:45.191740 7656 ts_manager.cc:194] Registered new tserver with Master: b5208d71fa9547f48092387d5a259a7a (127.3.243.130:35075)
I20260619 04:45:45.192708 7656 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:46473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:45:45.217666 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43333
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:45.538564 7984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:45.538895 7984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:45.539074 7984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:45.546000 7984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:45.546139 7984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:45.546341 7984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:45:45.555675 7984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43333
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:45.556915 7984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:45.558156 7984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:45.573330 7990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:45.574209 7989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:45.574959 7992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:45.575683 7984 server_base.cc:1034] running on GCE node
I20260619 04:45:45.576498 7984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:45.577766 7984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:45.579047 7984 hybrid_clock.cc:648] HybridClock initialized: now 1781844345578988 us; error 61 us; skew 500 ppm
I20260619 04:45:45.579509 7984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:45 dist-test-slave-2qp8 krb5kdc[7611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:45.583554 7984 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:45:45.585196 7984 webserver.cc:460] Webserver started at http://127.3.243.131:46605/ using document root <none> and password file <none>
I20260619 04:45:45.585886 7984 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:45.586027 7984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:45.586328 7984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:45.589411 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "1ff8467743f54073b5e0767d92060f87"
format_stamp: "Formatted at 2026-06-19 04:45:45 on dist-test-slave-2qp8"
server_key: "1248c1f1ad1bad2a26ec8a3bdcffb742"
server_key_iv: "f0943969bf08c0952a43769a861ad456"
server_key_version: "encryptionkey@0"
I20260619 04:45:45.590173 7984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "1ff8467743f54073b5e0767d92060f87"
format_stamp: "Formatted at 2026-06-19 04:45:45 on dist-test-slave-2qp8"
server_key: "1248c1f1ad1bad2a26ec8a3bdcffb742"
server_key_iv: "f0943969bf08c0952a43769a861ad456"
server_key_version: "encryptionkey@0"
I20260619 04:45:45.595963 7984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:45:45.599793 7999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:45.601423 7984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:45.601608 7984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "1ff8467743f54073b5e0767d92060f87"
format_stamp: "Formatted at 2026-06-19 04:45:45 on dist-test-slave-2qp8"
server_key: "1248c1f1ad1bad2a26ec8a3bdcffb742"
server_key_iv: "f0943969bf08c0952a43769a861ad456"
server_key_version: "encryptionkey@0"
I20260619 04:45:45.601816 7984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:45.644220 7984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:45.648661 7984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:45.648957 7984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:45.649916 7984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:45.651360 7984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:45.651460 7984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:45.651616 7984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:45.651685 7984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:45.695569 7984 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:38977
I20260619 04:45:45.695673 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:38977 every 8 connection(s)
I20260619 04:45:45.696969 7984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:45:45.704748 7845 heartbeater.cc:503] Master 127.3.243.190:43333 was elected leader, sending a full tablet report...
I20260619 04:45:45.706512 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 7984
I20260619 04:45:45.706743 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:45:45.707381 4046 external_mini_cluster.cc:1428] Setting key 3862ebdb873187000cc6a011f6d59d68
Jun 19 04:45:45 dist-test-slave-2qp8 krb5kdc[7611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844345, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:45.719707 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:45.699836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:36721 (local address 127.3.243.190:43333)
0619 04:45:45.700357 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:45.700362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:45.701568 (+ 1206us) server_negotiation.cc:407] Connection header received
0619 04:45:45.703457 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:45.703463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:45.703565 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:45.703691 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:45.705755 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.706869 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.708869 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.709329 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.709448 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:45.713869 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:45.713875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:45.713884 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:45.713915 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:45.716661 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.717373 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.717378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.717382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.717497 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.717923 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.717928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.717932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.718206 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:45.718359 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:45.719077 (+ 718us) server_negotiation.cc:299] Negotiation successful
0619 04:45:45.719301 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":310,"threads_started":1}
I20260619 04:45:45.720772 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:45.700624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:43333 (local address 127.3.243.131:36721)
0619 04:45:45.701401 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:45.701440 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:45.703170 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:45.703936 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:45.703945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:45.704673 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:45.705540 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:45.705565 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.706991 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.706996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:45.708682 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:45.708705 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.709749 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.709755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:45.710053 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:45.710832 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:45.710870 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:45.713652 (+ 2782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:45.716792 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:45.716801 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:45.716819 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:45.717245 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:45.717604 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:45.717609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:45.717613 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:45.717805 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:45.718339 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:45.718346 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:45.718676 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 04:45:45.718937 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 04:45:45.719462 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":329,"threads_started":1}
I20260619 04:45:45.722312 8112 heartbeater.cc:347] Connected to a master server at 127.3.243.190:43333
I20260619 04:45:45.722656 8112 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:45.723495 8112 heartbeater.cc:511] Master 127.3.243.190:43333 requested a full tablet report, sending...
I20260619 04:45:45.725050 7655 ts_manager.cc:194] Registered new tserver with Master: 1ff8467743f54073b5e0767d92060f87 (127.3.243.131:38977)
I20260619 04:45:45.726534 7655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:36721
I20260619 04:45:45.735693 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 04:45:45 dist-test-slave-2qp8 krb5kdc[7611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:45:45 dist-test-slave-2qp8 krb5kdc[7611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:45:45.787067 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:45.768376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.3.243.129:32885)
0619 04:45:45.769364 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:45.769376 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:45.769413 (+ 37us) server_negotiation.cc:407] Connection header received
0619 04:45:45.769496 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:45.769507 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:45.769826 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:45.770017 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:45.771412 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.773371 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.774552 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.775628 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.775773 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:45.779548 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:45.779566 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:45.779589 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:45.779642 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:45.783856 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.784601 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.784613 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.784619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.784791 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.785146 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.785153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.785158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.785684 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:45.786029 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:45.786217 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 04:45:45.786525 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":619,"threads_started":1}
Jun 19 04:45:45 dist-test-slave-2qp8 krb5kdc[7611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:45.809288 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:45.792325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.3.243.190:43333)
0619 04:45:45.792576 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:45.792584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:45.792729 (+ 145us) server_negotiation.cc:407] Connection header received
0619 04:45:45.792841 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:45.792849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:45.792992 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:45.793110 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:45.794388 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.795726 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.797023 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.797890 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.798075 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:45.802132 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:45.802141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:45.802149 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:45.802196 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:45.806335 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.806988 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.806997 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.807003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.807165 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.807494 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.807502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.807508 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.808040 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:45.808349 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:45.808563 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 04:45:45.808838 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
W20260619 04:45:45.810027 7655 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52638
I20260619 04:45:45.811260 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7717
I20260619 04:45:45.825497 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:45.812389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.3.243.129:32885)
0619 04:45:45.812603 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:45.812610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:45.812724 (+ 114us) server_negotiation.cc:407] Connection header received
0619 04:45:45.812831 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:45.812838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:45.812951 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:45.813066 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:45.814258 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.815510 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.816723 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.817574 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.817718 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:45.819913 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:45.819923 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:45.819930 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:45.819989 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:45.823016 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.823634 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.823646 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.823649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.823766 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.824190 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.824195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.824200 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.824440 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:45.824684 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:45.824969 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 04:45:45.825201 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 04:45:45.826092 7840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:45.912587 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 7717
I20260619 04:45:45.929497 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7851
Jun 19 04:45:45 dist-test-slave-2qp8 krb5kdc[7611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:45:45.944097 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:45.930404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.3.243.130:35075)
0619 04:45:45.930921 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:45.930928 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:45.930953 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:45:45.931018 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:45.931024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:45.931239 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:45.931365 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:45.932515 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.933454 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.934607 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:45.935140 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:45.935228 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:45.939012 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:45.939022 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:45.939036 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:45.939071 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:45.941754 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.942332 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.942338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.942340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.942459 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:45.942772 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:45.942775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:45.942776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:45.943067 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:45.943271 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:45.943513 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 04:45:45.943747 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":282,"threads_started":1}
I20260619 04:45:45.945343 7974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:46.027218 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 7851
I20260619 04:45:46.042732 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7984
Jun 19 04:45:46 dist-test-slave-2qp8 krb5kdc[7611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:45:46.059095 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:46.043725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.3.243.131:38977)
0619 04:45:46.044432 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:46.044438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:46.044467 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:45:46.044528 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:46.044534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:46.044743 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:46.044912 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:46.046187 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:46.047176 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:46.048522 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:46.049079 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:46.049218 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:46.053384 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:46.053394 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:46.053408 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:46.053439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:46.056412 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:46.057070 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:46.057077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:46.057079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:46.057202 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:46.057576 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:46.057583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:46.057586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:46.057915 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:46.058258 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:46.058488 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 04:45:46.058753 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":395,"threads_started":1}
I20260619 04:45:46.060390 8107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:46.142326 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 7984
I20260619 04:45:46.157766 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 7624
I20260619 04:45:46.158926 7686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:46.266466 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 7624
2026-06-19T04:45:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4002 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:46 dist-test-slave-2qp8 krb5kdc[8132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:46 dist-test-slave-2qp8 krb5kdc[8132](info): set up 2 sockets
Jun 19 04:45:46 dist-test-slave-2qp8 krb5kdc[8132](info): commencing operation
krb5kdc: starting...
W20260619 04:45:47.344720 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:47 dist-test-slave-2qp8 krb5kdc[8132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:47.537815 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:37029
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:37029
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:47.853111 8145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:47.853433 8145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:47.853652 8145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:47.860735 8145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:47.860827 8145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:47.860935 8145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:47.861059 8145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:47.861146 8145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:47.870784 8145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:37029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:37029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:47.872045 8145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:47.873347 8145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:47.885010 8150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:47.884977 8153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:47.885056 8151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:47.886207 8145 server_base.cc:1034] running on GCE node
I20260619 04:45:47.886904 8145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:47.888270 8145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:47.889473 8145 hybrid_clock.cc:648] HybridClock initialized: now 1781844347889450 us; error 48 us; skew 500 ppm
I20260619 04:45:47.889829 8145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:47 dist-test-slave-2qp8 krb5kdc[8132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844347, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:47.892794 8145 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:47.893918 8145 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:47.894058 8145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:47.894388 8145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:47.897487 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "5cfa8b72a33f444289af6f2ffffe8b28"
format_stamp: "Formatted at 2026-06-19 04:45:47 on dist-test-slave-2qp8"
server_key: "5a9a8d886993e653910e572e2e00e25a"
server_key_iv: "f71f1ae2106565afe24979cd30f3d4a8"
server_key_version: "encryptionkey@0"
I20260619 04:45:47.898252 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "5cfa8b72a33f444289af6f2ffffe8b28"
format_stamp: "Formatted at 2026-06-19 04:45:47 on dist-test-slave-2qp8"
server_key: "5a9a8d886993e653910e572e2e00e25a"
server_key_iv: "f71f1ae2106565afe24979cd30f3d4a8"
server_key_version: "encryptionkey@0"
I20260619 04:45:47.903183 8145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:45:47.906550 8159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:47.907863 8145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:47.908056 8145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "5cfa8b72a33f444289af6f2ffffe8b28"
format_stamp: "Formatted at 2026-06-19 04:45:47 on dist-test-slave-2qp8"
server_key: "5a9a8d886993e653910e572e2e00e25a"
server_key_iv: "f71f1ae2106565afe24979cd30f3d4a8"
server_key_version: "encryptionkey@0"
I20260619 04:45:47.908286 8145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:47.923964 8145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:47.929143 8145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:47.929414 8145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:47.947180 8145 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:37029
I20260619 04:45:47.947216 8210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:37029 every 8 connection(s)
I20260619 04:45:47.948086 8145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:47.951809 8211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:47.956224 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 8145
I20260619 04:45:47.956445 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:47.957099 4046 external_mini_cluster.cc:1428] Setting key 70b0a7a243b9cc79bb247d04042ac870
I20260619 04:45:47.963860 8211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28: Bootstrap starting.
I20260619 04:45:47.967629 8211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:47.968832 8211 log.cc:824] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:47.971628 8211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28: No bootstrap required, opened a new log
Jun 19 04:45:47 dist-test-slave-2qp8 krb5kdc[8132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:47.981882 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:47.959098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37782 (local address 127.3.243.190:37029)
0619 04:45:47.959913 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:47.959934 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:47.960001 (+ 67us) server_negotiation.cc:407] Connection header received
0619 04:45:47.962453 (+ 2452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:47.962481 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:47.962986 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:47.963404 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:47.964796 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:47.966868 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:47.968158 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:47.969166 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:47.969375 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:47.973267 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:47.973277 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:47.973296 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:47.973347 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:47.977416 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:47.978086 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:47.978095 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:47.978120 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:47.978358 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:47.978676 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:47.978681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:47.978685 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:47.979384 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:47.979688 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:47.979976 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 04:45:47.980498 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":326,"threads_started":1}
I20260619 04:45:47.982416 8211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfa8b72a33f444289af6f2ffffe8b28" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37029 } }
I20260619 04:45:47.982801 8211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:47.982913 8211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfa8b72a33f444289af6f2ffffe8b28, State: Initialized, Role: FOLLOWER
I20260619 04:45:47.983552 8211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfa8b72a33f444289af6f2ffffe8b28" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37029 } }
I20260619 04:45:47.983789 8211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:47.983939 8211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:47.984180 8211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:47.986444 8211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfa8b72a33f444289af6f2ffffe8b28" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37029 } }
I20260619 04:45:47.986922 8211 leader_election.cc:302] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cfa8b72a33f444289af6f2ffffe8b28; no voters:
I20260619 04:45:47.987488 8211 leader_election.cc:288] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:47.987636 8216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:47.988314 8216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [term 1 LEADER]: Becoming Leader. State: Replica: 5cfa8b72a33f444289af6f2ffffe8b28, State: Running, Role: LEADER
I20260619 04:45:47.988845 8216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfa8b72a33f444289af6f2ffffe8b28" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37029 } }
I20260619 04:45:47.989600 8211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:47.991379 8218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cfa8b72a33f444289af6f2ffffe8b28. Latest consensus state: current_term: 1 leader_uuid: "5cfa8b72a33f444289af6f2ffffe8b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfa8b72a33f444289af6f2ffffe8b28" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37029 } } }
I20260619 04:45:47.991744 8218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:47.992038 8217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cfa8b72a33f444289af6f2ffffe8b28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cfa8b72a33f444289af6f2ffffe8b28" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37029 } } }
I20260619 04:45:47.992523 8217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:48.001636 8221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:48.006671 8221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:48.015990 8221 catalog_manager.cc:1347] Generated new cluster ID: a7b08ea1e5644f328eed4db23a69e5b9
I20260619 04:45:48.016131 8221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:48.030975 8221 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:48.032542 8221 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:48.039570 8221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28: Generated new TSK 0
I20260619 04:45:48.040532 8221 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:45:48.095578 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37029
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:48.438238 8238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:48.438529 8238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:48.438722 8238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:48.445982 8238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:48.446115 8238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:48.446303 8238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:48.455922 8238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:48.457231 8238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:48.458534 8238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:48.471189 8243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:48.471109 8244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:48.471144 8246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:48.472260 8238 server_base.cc:1034] running on GCE node
I20260619 04:45:48.472823 8238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:48.473685 8238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:48.474876 8238 hybrid_clock.cc:648] HybridClock initialized: now 1781844348474807 us; error 61 us; skew 500 ppm
I20260619 04:45:48.475150 8238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:48 dist-test-slave-2qp8 krb5kdc[8132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:48.478370 8238 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:45:48.479504 8238 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:48.479594 8238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:48.479848 8238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:48.482818 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "be01c9e4c0564393832b7381ae6c80ad"
format_stamp: "Formatted at 2026-06-19 04:45:48 on dist-test-slave-2qp8"
server_key: "bce3d969974c034b01258b04c3670c23"
server_key_iv: "0423e212265d9b00d9912958901851e5"
server_key_version: "encryptionkey@0"
I20260619 04:45:48.483639 8238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "be01c9e4c0564393832b7381ae6c80ad"
format_stamp: "Formatted at 2026-06-19 04:45:48 on dist-test-slave-2qp8"
server_key: "bce3d969974c034b01258b04c3670c23"
server_key_iv: "0423e212265d9b00d9912958901851e5"
server_key_version: "encryptionkey@0"
I20260619 04:45:48.488631 8238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 04:45:48.492147 8252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:48.493433 8238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 04:45:48.493602 8238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "be01c9e4c0564393832b7381ae6c80ad"
format_stamp: "Formatted at 2026-06-19 04:45:48 on dist-test-slave-2qp8"
server_key: "bce3d969974c034b01258b04c3670c23"
server_key_iv: "0423e212265d9b00d9912958901851e5"
server_key_version: "encryptionkey@0"
I20260619 04:45:48.493808 8238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:48.511338 8238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:48.515543 8238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:48.515820 8238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:48.516791 8238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:48.518059 8238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:48.518177 8238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:48.518303 8238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:48.518399 8238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:48.550861 8238 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:36451
I20260619 04:45:48.550911 8364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:36451 every 8 connection(s)
I20260619 04:45:48.551740 8238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:48.555661 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 8238
I20260619 04:45:48.555886 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:48.556665 4046 external_mini_cluster.cc:1428] Setting key 96c9f343bd6629612b0fa12ee94d2609
Jun 19 04:45:48 dist-test-slave-2qp8 krb5kdc[8132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:48.574100 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:48.554557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:39075 (local address 127.3.243.190:37029)
0619 04:45:48.554978 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:48.554985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:48.556235 (+ 1250us) server_negotiation.cc:407] Connection header received
0619 04:45:48.558493 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:48.558497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:48.558558 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:48.558681 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:48.560162 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:48.561320 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:48.562882 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:48.563603 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:48.563745 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:48.567852 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:48.567857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:48.567861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:48.567900 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:48.570527 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:48.571276 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:48.571288 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:48.571290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:48.571405 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:48.571875 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:48.571884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:48.571891 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:48.572344 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:48.572525 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:48.573367 (+ 842us) server_negotiation.cc:299] Negotiation successful
0619 04:45:48.573654 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":205,"threads_started":1}
I20260619 04:45:48.575098 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:48.555126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37029 (local address 127.3.243.129:39075)
0619 04:45:48.556038 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:48.556086 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:48.558170 (+ 2084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:48.558878 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:48.558886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:48.559417 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:48.559948 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:48.559969 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:48.561474 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:48.561478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:48.562727 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:48.562739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:48.563836 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:48.563841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:48.564041 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:48.564878 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:48.564913 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:48.567605 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:48.570676 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:48.570686 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:48.570703 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:48.571145 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:48.571519 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:48.571525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:48.571528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:48.571717 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:48.572530 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:48.572538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:48.572916 (+ 378us) client_negotiation.cc:769] Sending connection context
0619 04:45:48.573212 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 04:45:48.573696 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":326,"threads_started":1}
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:45:48.576956 8365 heartbeater.cc:347] Connected to a master server at 127.3.243.190:37029
I20260619 04:45:48.577307 8365 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:48.578038 8365 heartbeater.cc:511] Master 127.3.243.190:37029 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
I20260619 04:45:48.580917 8176 ts_manager.cc:194] Registered new tserver with Master: be01c9e4c0564393832b7381ae6c80ad (127.3.243.129:36451)
I20260619 04:45:48.583300 8176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:39075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:45:48.605239 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37029
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:48.919248 8371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:48.919586 8371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:48.919780 8371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:48.927021 8371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:48.927207 8371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:48.927405 8371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:48.938644 8371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:48.940050 8371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:48.941440 8371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:48.954200 8377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:48.954231 8376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:48.954876 8379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:48.955545 8371 server_base.cc:1034] running on GCE node
I20260619 04:45:48.956180 8371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:48.957129 8371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:48.958352 8371 hybrid_clock.cc:648] HybridClock initialized: now 1781844348958297 us; error 49 us; skew 500 ppm
I20260619 04:45:48.958664 8371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:48 dist-test-slave-2qp8 krb5kdc[8132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:48.961913 8371 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:45:48.963197 8371 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:48.963328 8371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:48.963637 8371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:48.966495 8371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "709a7907ecbd4eeca8fe29703eb0339f"
format_stamp: "Formatted at 2026-06-19 04:45:48 on dist-test-slave-2qp8"
server_key: "a2f8066b9cf5b05176ee550dfa916261"
server_key_iv: "d9ffdfc5050e8bd65309eab795d3740c"
server_key_version: "encryptionkey@0"
I20260619 04:45:48.967326 8371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "709a7907ecbd4eeca8fe29703eb0339f"
format_stamp: "Formatted at 2026-06-19 04:45:48 on dist-test-slave-2qp8"
server_key: "a2f8066b9cf5b05176ee550dfa916261"
server_key_iv: "d9ffdfc5050e8bd65309eab795d3740c"
server_key_version: "encryptionkey@0"
I20260619 04:45:48.972209 8371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 04:45:48.975492 8385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:48.976758 8371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 04:45:48.976931 8371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "709a7907ecbd4eeca8fe29703eb0339f"
format_stamp: "Formatted at 2026-06-19 04:45:48 on dist-test-slave-2qp8"
server_key: "a2f8066b9cf5b05176ee550dfa916261"
server_key_iv: "d9ffdfc5050e8bd65309eab795d3740c"
server_key_version: "encryptionkey@0"
I20260619 04:45:48.977187 8371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:49.018190 8371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:49.022223 8371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:49.022512 8371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:49.023551 8371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:49.025020 8371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:49.025143 8371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:49.025281 8371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:49.025367 8371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:49.057842 8371 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:42593
I20260619 04:45:49.057878 8497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:42593 every 8 connection(s)
I20260619 04:45:49.058761 8371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:49.064420 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 8371
I20260619 04:45:49.064630 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:49.065069 4046 external_mini_cluster.cc:1428] Setting key 88d22c41b6df9a7b5cc47f27d0bb484b
Jun 19 04:45:49 dist-test-slave-2qp8 krb5kdc[8132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844348, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:49.079433 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.061307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:41505 (local address 127.3.243.190:37029)
0619 04:45:49.061488 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.061492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.062698 (+ 1206us) server_negotiation.cc:407] Connection header received
0619 04:45:49.064601 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.064607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.064698 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.064822 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:49.066598 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.067595 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.069209 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.069952 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.070096 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.073640 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:49.073645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:49.073648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:49.073680 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:49.076499 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.077216 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.077225 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.077227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.077340 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.077767 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.077772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.077775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.078041 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:49.078191 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.078871 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.079102 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:45:49.080595 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.061836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37029 (local address 127.3.243.130:41505)
0619 04:45:49.062538 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:49.062576 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:49.064307 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:49.065053 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:49.065062 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:49.065691 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:49.066421 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.066445 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.067744 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.067748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.069057 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.069073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.070149 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.070153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.070337 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.071037 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:49.071058 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:49.073401 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:49.076632 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:49.076642 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:49.076659 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:49.077089 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:49.077458 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:49.077471 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:49.077475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:49.077649 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:49.078168 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:49.078176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:49.078511 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 04:45:49.078777 (+ 266us) client_negotiation.cc:240] Negotiation successful
0619 04:45:49.079297 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":280,"threads_started":1}
I20260619 04:45:49.082082 8498 heartbeater.cc:347] Connected to a master server at 127.3.243.190:37029
I20260619 04:45:49.082387 8498 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:49.083034 8498 heartbeater.cc:511] Master 127.3.243.190:37029 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:45:49.084436 8176 ts_manager.cc:194] Registered new tserver with Master: 709a7907ecbd4eeca8fe29703eb0339f (127.3.243.130:42593)
I20260619 04:45:49.085249 8176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:41505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:45:49.109575 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37029
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:49.423358 8503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:49.423661 8503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:49.423754 8503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:49.431039 8503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:49.431154 8503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:49.431273 8503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:45:49.441084 8503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:37029
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:49.442345 8503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:49.443619 8503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:49.456538 8509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:49.456630 8508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:49.456502 8511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:49.457403 8503 server_base.cc:1034] running on GCE node
I20260619 04:45:49.458006 8503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:49.459064 8503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:49.460281 8503 hybrid_clock.cc:648] HybridClock initialized: now 1781844349460244 us; error 60 us; skew 500 ppm
I20260619 04:45:49.460670 8503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:49 dist-test-slave-2qp8 krb5kdc[8132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:49.463862 8503 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:45:49.465121 8503 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:49.465215 8503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:49.465485 8503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:49.468604 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "593098fa25e34b26969f161d081507de"
format_stamp: "Formatted at 2026-06-19 04:45:49 on dist-test-slave-2qp8"
server_key: "3759e7d433c0c40d05bc1688a41b7f78"
server_key_iv: "73ec7fb1efca4a0395faeee921659e51"
server_key_version: "encryptionkey@0"
I20260619 04:45:49.469388 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "593098fa25e34b26969f161d081507de"
format_stamp: "Formatted at 2026-06-19 04:45:49 on dist-test-slave-2qp8"
server_key: "3759e7d433c0c40d05bc1688a41b7f78"
server_key_iv: "73ec7fb1efca4a0395faeee921659e51"
server_key_version: "encryptionkey@0"
I20260619 04:45:49.474606 8503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 04:45:49.478456 8517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:49.480010 8503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:49.480278 8503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "593098fa25e34b26969f161d081507de"
format_stamp: "Formatted at 2026-06-19 04:45:49 on dist-test-slave-2qp8"
server_key: "3759e7d433c0c40d05bc1688a41b7f78"
server_key_iv: "73ec7fb1efca4a0395faeee921659e51"
server_key_version: "encryptionkey@0"
I20260619 04:45:49.480566 8503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:49.504670 8503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:49.509753 8503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:49.510010 8503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:49.511037 8503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:49.512518 8503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:49.512616 8503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:49.512702 8503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:49.512763 8503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:49.546898 8503 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:39105
I20260619 04:45:49.546941 8629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:39105 every 8 connection(s)
I20260619 04:45:49.547813 8503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:45:49.548666 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 8503
I20260619 04:45:49.549188 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:45:49.550034 4046 external_mini_cluster.cc:1428] Setting key 1d73cdfe19eaee272f963ca28e315552
Jun 19 04:45:49 dist-test-slave-2qp8 krb5kdc[8132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844349, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:49.570785 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.550851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:49219 (local address 127.3.243.190:37029)
0619 04:45:49.551062 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.551068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.553599 (+ 2531us) server_negotiation.cc:407] Connection header received
0619 04:45:49.555668 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.555673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.555767 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.555903 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:49.557548 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.558435 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.560076 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.560617 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.560769 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.564394 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:49.564401 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:49.564407 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:49.564453 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:49.567553 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.568422 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.568427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.568430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.568561 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.569021 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.569032 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.569034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.569302 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:49.569475 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.570178 (+ 703us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.570407 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 04:45:49.571770 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.551361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37029 (local address 127.3.243.131:49219)
0619 04:45:49.553404 (+ 2043us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:49.553453 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:49.555381 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:49.556185 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:49.556197 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:49.556749 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:49.557317 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.557392 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.558602 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.558608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.559911 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.559922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.560857 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.560868 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.561122 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.561737 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:49.561761 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:49.564166 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:49.567704 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:49.567713 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:49.567740 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:49.568283 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:49.568682 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:49.568689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:49.568693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:49.568888 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:49.569446 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:49.569454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:49.569803 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 04:45:49.570070 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 04:45:49.570635 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1727,"thread_start_us":667,"threads_started":1}
I20260619 04:45:49.573083 8630 heartbeater.cc:347] Connected to a master server at 127.3.243.190:37029
I20260619 04:45:49.573376 8630 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:49.573990 8630 heartbeater.cc:511] Master 127.3.243.190:37029 requested a full tablet report, sending...
I20260619 04:45:49.575310 8176 ts_manager.cc:194] Registered new tserver with Master: 593098fa25e34b26969f161d081507de (127.3.243.131:39105)
I20260619 04:45:49.576085 8176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:49219
I20260619 04:45:49.578006 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:45:49.586521 8365 heartbeater.cc:503] Master 127.3.243.190:37029 was elected leader, sending a full tablet report...
I20260619 04:45:49.596457 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.582700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37788 (local address 127.3.243.190:37029)
0619 04:45:49.582878 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.582882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.583470 (+ 588us) server_negotiation.cc:407] Connection header received
0619 04:45:49.583613 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.583617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.583704 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.583804 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:49.585507 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.586186 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.587778 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.588474 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.588625 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.591159 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:49.591166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:49.591172 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:49.591215 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:49.593807 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.594447 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.594454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.594458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.594602 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.595014 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.595022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.595026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.595326 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:49.595606 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.595902 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.596137 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:45:49.600662 8176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:45:49.607686 8176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:49.631753 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.619472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:42593 (local address 127.0.0.1:50162)
0619 04:45:49.620343 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:49.620387 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:49.620563 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:49.621982 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:49.621993 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:49.622034 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:49.622495 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.622511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.625064 (+ 2553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.625073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.627964 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.627991 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.630374 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.630387 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.630697 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.630777 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 04:45:49.630914 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 04:45:49.631176 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":362,"threads_started":1}
I20260619 04:45:49.632017 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.619674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.3.243.130:42593)
0619 04:45:49.621237 (+ 1563us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.621250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.621283 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:45:49.621400 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.621411 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.621705 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.622746 (+ 1041us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:49.623005 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.624877 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.628245 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.630078 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.631214 (+ 1136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.631436 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.631537 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.631715 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1348,"thread_start_us":1832,"threads_started":1}
I20260619 04:45:49.636003 8433 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf5745ad56a42f8b820c641fd60f7a4 (DEFAULT_TABLE table=test-table [id=473e93b43a6941f8a0ca6e1dc22a6225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:49.637562 8433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf5745ad56a42f8b820c641fd60f7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:49.638733 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.620833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:39105 (local address 127.0.0.1:55114)
0619 04:45:49.630392 (+ 9559us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:49.630407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:49.630531 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:49.631267 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:49.631271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:49.631295 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:49.631673 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.631685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.634100 (+ 2415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.634108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.636167 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.636181 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.637984 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.637992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.638156 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.638187 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:45:49.638288 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 04:45:49.638406 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9420,"thread_start_us":9303,"threads_started":1}
I20260619 04:45:49.638828 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.621213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.3.243.131:39105)
0619 04:45:49.629450 (+ 8237us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.629459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.630658 (+ 1199us) server_negotiation.cc:407] Connection header received
0619 04:45:49.630763 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.630773 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.631061 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.631876 (+ 815us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:49.632321 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.633940 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.636367 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.637773 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.637903 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.638148 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.638342 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.638500 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8037,"thread_start_us":7869,"threads_started":1}
I20260619 04:45:49.640522 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.620208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:36451 (local address 127.0.0.1:49422)
0619 04:45:49.629699 (+ 9491us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:49.629723 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:49.629863 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:49.633340 (+ 3477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:49.633345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:49.633370 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:49.633755 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.633765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.635922 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.635928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.637979 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.637993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.639793 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.639802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.639965 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.639994 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 04:45:49.640082 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 04:45:49.640214 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9333,"thread_start_us":9197,"threads_started":1}
I20260619 04:45:49.640633 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.620818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49422 (local address 127.3.243.129:36451)
0619 04:45:49.632736 (+ 11918us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.632745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.632772 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:45:49.632862 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.632869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.633129 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.633955 (+ 826us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:49.634219 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.635773 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.638153 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.639578 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.639721 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.639938 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.640140 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.640289 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11731,"thread_start_us":11560,"threads_started":1}
I20260619 04:45:49.642510 8565 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf5745ad56a42f8b820c641fd60f7a4 (DEFAULT_TABLE table=test-table [id=473e93b43a6941f8a0ca6e1dc22a6225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:49.643863 8565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf5745ad56a42f8b820c641fd60f7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:49.644038 8300 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf5745ad56a42f8b820c641fd60f7a4 (DEFAULT_TABLE table=test-table [id=473e93b43a6941f8a0ca6e1dc22a6225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:49.645380 8300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf5745ad56a42f8b820c641fd60f7a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:49.650707 8648 tablet_bootstrap.cc:492] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: Bootstrap starting.
I20260619 04:45:49.653853 8648 tablet_bootstrap.cc:654] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:49.654795 8648 log.cc:824] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:49.657249 8648 tablet_bootstrap.cc:492] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: No bootstrap required, opened a new log
I20260619 04:45:49.657572 8648 ts_tablet_manager.cc:1397] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 04:45:49.658178 8650 tablet_bootstrap.cc:492] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: Bootstrap starting.
I20260619 04:45:49.660573 8651 tablet_bootstrap.cc:492] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: Bootstrap starting.
I20260619 04:45:49.662132 8650 tablet_bootstrap.cc:654] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:49.663177 8651 tablet_bootstrap.cc:654] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:49.663339 8650 log.cc:824] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:49.664366 8651 log.cc:824] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:49.666381 8650 tablet_bootstrap.cc:492] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: No bootstrap required, opened a new log
I20260619 04:45:49.666639 8650 ts_tablet_manager.cc:1397] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 04:45:49.667953 8648 raft_consensus.cc:348] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } }
I20260619 04:45:49.668406 8648 raft_consensus.cc:374] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:49.668512 8648 raft_consensus.cc:729] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709a7907ecbd4eeca8fe29703eb0339f, State: Initialized, Role: FOLLOWER
I20260619 04:45:49.669178 8648 consensus_queue.cc:260] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } }
I20260619 04:45:49.669526 8651 tablet_bootstrap.cc:492] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: No bootstrap required, opened a new log
I20260619 04:45:49.669751 8651 ts_tablet_manager.cc:1397] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 04:45:49.670518 8498 heartbeater.cc:503] Master 127.3.243.190:37029 was elected leader, sending a full tablet report...
I20260619 04:45:49.671108 8648 ts_tablet_manager.cc:1428] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 04:45:49.676381 8650 raft_consensus.cc:348] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } }
I20260619 04:45:49.676858 8650 raft_consensus.cc:374] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:49.676973 8650 raft_consensus.cc:729] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593098fa25e34b26969f161d081507de, State: Initialized, Role: FOLLOWER
I20260619 04:45:49.677560 8650 consensus_queue.cc:260] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } }
I20260619 04:45:49.678576 8630 heartbeater.cc:503] Master 127.3.243.190:37029 was elected leader, sending a full tablet report...
I20260619 04:45:49.678455 8651 raft_consensus.cc:348] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } }
I20260619 04:45:49.678870 8651 raft_consensus.cc:374] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:49.678977 8651 raft_consensus.cc:729] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be01c9e4c0564393832b7381ae6c80ad, State: Initialized, Role: FOLLOWER
I20260619 04:45:49.679306 8650 ts_tablet_manager.cc:1428] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 04:45:49.679541 8651 consensus_queue.cc:260] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } }
I20260619 04:45:49.682250 8651 ts_tablet_manager.cc:1428] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260619 04:45:49.698290 8656 raft_consensus.cc:482] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:49.698529 8656 raft_consensus.cc:504] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } }
I20260619 04:45:49.700074 8656 leader_election.cc:288] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 709a7907ecbd4eeca8fe29703eb0339f (127.3.243.130:42593), 593098fa25e34b26969f161d081507de (127.3.243.131:39105)
I20260619 04:45:49.706816 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.700520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:48751 (local address 127.3.243.130:42593)
0619 04:45:49.700706 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.700711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.701181 (+ 470us) server_negotiation.cc:407] Connection header received
0619 04:45:49.701484 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.701494 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.701602 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.701665 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:49.702380 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.703151 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.705043 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.705855 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.706046 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.706123 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.706390 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.706518 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:45:49.706939 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.700432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:42593 (local address 127.3.243.129:48751)
0619 04:45:49.701077 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:49.701098 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:49.701232 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:49.701860 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:49.701863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:49.701883 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:49.702236 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.702244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.703283 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.703291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.704853 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.704865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.706055 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.706061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.706231 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.706255 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 04:45:49.706362 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 04:45:49.706511 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":4,"spinlock_wait_cycles":16768,"thread_start_us":357,"threads_started":1}
I20260619 04:45:49.707290 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.701372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:47571 (local address 127.3.243.131:39105)
0619 04:45:49.701594 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.701602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.701937 (+ 335us) server_negotiation.cc:407] Connection header received
0619 04:45:49.702161 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.702166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.702230 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.702324 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:45:49.702895 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.703587 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.705542 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.706302 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.706486 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.706568 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.706899 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.707048 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:45:49.707942 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.701219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:39105 (local address 127.3.243.129:47571)
0619 04:45:49.701834 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:49.701848 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:49.702025 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:49.702340 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:49.702343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:49.702366 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:45:49.702750 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.702760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.703732 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.703738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.705370 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:49.705382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.706520 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.706526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:49.706714 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.706743 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 04:45:49.707459 (+ 716us) client_negotiation.cc:240] Negotiation successful
0619 04:45:49.707584 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":17,"thread_start_us":359,"threads_started":1}
I20260619 04:45:49.707979 8453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5745ad56a42f8b820c641fd60f7a4" candidate_uuid: "be01c9e4c0564393832b7381ae6c80ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709a7907ecbd4eeca8fe29703eb0339f" is_pre_election: true
I20260619 04:45:49.708592 8453 raft_consensus.cc:2454] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be01c9e4c0564393832b7381ae6c80ad in term 0.
I20260619 04:45:49.709021 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5745ad56a42f8b820c641fd60f7a4" candidate_uuid: "be01c9e4c0564393832b7381ae6c80ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593098fa25e34b26969f161d081507de" is_pre_election: true
I20260619 04:45:49.709218 8255 leader_election.cc:302] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 709a7907ecbd4eeca8fe29703eb0339f, be01c9e4c0564393832b7381ae6c80ad; no voters:
I20260619 04:45:49.709501 8585 raft_consensus.cc:2454] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be01c9e4c0564393832b7381ae6c80ad in term 0.
I20260619 04:45:49.709601 8656 raft_consensus.cc:2781] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:49.709753 8656 raft_consensus.cc:482] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:49.709885 8656 raft_consensus.cc:3037] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:49.712249 8656 raft_consensus.cc:504] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } }
I20260619 04:45:49.712775 8656 leader_election.cc:288] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [CANDIDATE]: Term 1 election: Requested vote from peers 709a7907ecbd4eeca8fe29703eb0339f (127.3.243.130:42593), 593098fa25e34b26969f161d081507de (127.3.243.131:39105)
I20260619 04:45:49.713198 8453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5745ad56a42f8b820c641fd60f7a4" candidate_uuid: "be01c9e4c0564393832b7381ae6c80ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709a7907ecbd4eeca8fe29703eb0339f"
I20260619 04:45:49.713320 8585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5745ad56a42f8b820c641fd60f7a4" candidate_uuid: "be01c9e4c0564393832b7381ae6c80ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593098fa25e34b26969f161d081507de"
I20260619 04:45:49.713397 8453 raft_consensus.cc:3037] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:49.713528 8585 raft_consensus.cc:3037] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:49.715477 8453 raft_consensus.cc:2454] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be01c9e4c0564393832b7381ae6c80ad in term 1.
I20260619 04:45:49.715595 8585 raft_consensus.cc:2454] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be01c9e4c0564393832b7381ae6c80ad in term 1.
I20260619 04:45:49.715885 8255 leader_election.cc:302] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 709a7907ecbd4eeca8fe29703eb0339f, be01c9e4c0564393832b7381ae6c80ad; no voters:
I20260619 04:45:49.716223 8656 raft_consensus.cc:2781] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:49.716598 8656 raft_consensus.cc:686] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 1 LEADER]: Becoming Leader. State: Replica: be01c9e4c0564393832b7381ae6c80ad, State: Running, Role: LEADER
I20260619 04:45:49.717167 8656 consensus_queue.cc:237] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } }
I20260619 04:45:49.721659 8175 catalog_manager.cc:5495] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad reported cstate change: term changed from 0 to 1, leader changed from <none> to be01c9e4c0564393832b7381ae6c80ad (127.3.243.129). New cstate: current_term: 1 leader_uuid: "be01c9e4c0564393832b7381ae6c80ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be01c9e4c0564393832b7381ae6c80ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 36451 } health_report { overall_health: HEALTHY } } }
I20260619 04:45:49.738503 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.733335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.3.243.129:36451)
0619 04:45:49.733543 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.733548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.733566 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:45:49.733756 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.733759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.733838 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.733985 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:45:49.734340 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.735045 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.736458 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.736871 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.737001 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.737451 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:45:49.737923 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:45:49.738081 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.738139 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.738226 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:45:49.746105 8585 raft_consensus.cc:1261] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [term 1 FOLLOWER]: Refusing update from remote peer be01c9e4c0564393832b7381ae6c80ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:49.746138 8453 raft_consensus.cc:1261] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [term 1 FOLLOWER]: Refusing update from remote peer be01c9e4c0564393832b7381ae6c80ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:49.747030 8659 consensus_queue.cc:1035] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "709a7907ecbd4eeca8fe29703eb0339f" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:49.747431 8656 consensus_queue.cc:1035] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "593098fa25e34b26969f161d081507de" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:49.773396 8176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:49.773676 8176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:49.777050 8176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cfa8b72a33f444289af6f2ffffe8b28: Sending DeleteTablet for 3 replicas of tablet 0bf5745ad56a42f8b820c641fd60f7a4
I20260619 04:45:49.778227 8300 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf5745ad56a42f8b820c641fd60f7a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:49 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:49422
I20260619 04:45:49.778220 8565 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf5745ad56a42f8b820c641fd60f7a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:49 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:55114
I20260619 04:45:49.778261 8433 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf5745ad56a42f8b820c641fd60f7a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:49 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:50162
I20260619 04:45:49.780576 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 8238
I20260619 04:45:49.789172 8673 tablet_replica.cc:330] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: stopping tablet replica
I20260619 04:45:49.790511 8673 raft_consensus.cc:2229] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:49.791977 8673 raft_consensus.cc:2258] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:49.792660 8672 tablet_replica.cc:330] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: stopping tablet replica
I20260619 04:45:49.792840 8671 tablet_replica.cc:330] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: stopping tablet replica
I20260619 04:45:49.793260 8671 raft_consensus.cc:2229] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:49.793062 8672 raft_consensus.cc:2229] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:45:49.793776 8671 raft_consensus.cc:2258] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:49.794212 8672 raft_consensus.cc:2258] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:49.796401 8673 ts_tablet_manager.cc:1905] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:49.796438 8671 ts_tablet_manager.cc:1905] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:49.796746 8672 ts_tablet_manager.cc:1905] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:49.804152 8671 ts_tablet_manager.cc:1918] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:45:49.804317 8671 log.cc:1196] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/0bf5745ad56a42f8b820c641fd60f7a4
I20260619 04:45:49.804901 8671 ts_tablet_manager.cc:1939] T 0bf5745ad56a42f8b820c641fd60f7a4 P 593098fa25e34b26969f161d081507de: Deleting consensus metadata
I20260619 04:45:49.805644 8673 ts_tablet_manager.cc:1918] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:45:49.805860 8673 log.cc:1196] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/0bf5745ad56a42f8b820c641fd60f7a4
I20260619 04:45:49.806483 8673 ts_tablet_manager.cc:1939] T 0bf5745ad56a42f8b820c641fd60f7a4 P 709a7907ecbd4eeca8fe29703eb0339f: Deleting consensus metadata
I20260619 04:45:49.806841 8162 catalog_manager.cc:4841] TS 593098fa25e34b26969f161d081507de (127.3.243.131:39105): tablet 0bf5745ad56a42f8b820c641fd60f7a4 (table test-table [id=473e93b43a6941f8a0ca6e1dc22a6225]) successfully deleted
Jun 19 04:45:49 dist-test-slave-2qp8 krb5kdc[8132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:45:49.808225 8162 catalog_manager.cc:4841] TS 709a7907ecbd4eeca8fe29703eb0339f (127.3.243.130:42593): tablet 0bf5745ad56a42f8b820c641fd60f7a4 (table test-table [id=473e93b43a6941f8a0ca6e1dc22a6225]) successfully deleted
I20260619 04:45:49.809358 8672 ts_tablet_manager.cc:1918] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:45:49.809652 8672 log.cc:1196] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/0bf5745ad56a42f8b820c641fd60f7a4
I20260619 04:45:49.810231 8672 ts_tablet_manager.cc:1939] T 0bf5745ad56a42f8b820c641fd60f7a4 P be01c9e4c0564393832b7381ae6c80ad: Deleting consensus metadata
I20260619 04:45:49.811877 8163 catalog_manager.cc:4841] TS be01c9e4c0564393832b7381ae6c80ad (127.3.243.129:36451): tablet 0bf5745ad56a42f8b820c641fd60f7a4 (table test-table [id=473e93b43a6941f8a0ca6e1dc22a6225]) successfully deleted
I20260619 04:45:49.815774 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.781969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.3.243.129:36451)
0619 04:45:49.782181 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.782190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.791747 (+ 9557us) server_negotiation.cc:407] Connection header received
0619 04:45:49.791966 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.791973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.792052 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.792187 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:49.796002 (+ 3815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.797122 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.798935 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.799649 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.799745 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.809197 (+ 9452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:49.809212 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:49.809232 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:49.809282 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:49.813047 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.813873 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.813880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.813883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.814038 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.814422 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.814425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.814427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.814734 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:49.814931 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.815289 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.815477 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:45:49.816529 8360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:49.913401 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 8238
I20260619 04:45:49.931945 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 8371
Jun 19 04:45:49 dist-test-slave-2qp8 krb5kdc[8132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:45:49.947271 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:49.932928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.3.243.130:42593)
0619 04:45:49.933118 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:49.933124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:49.933651 (+ 527us) server_negotiation.cc:407] Connection header received
0619 04:45:49.933731 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:49.933737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:49.933836 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:49.933946 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:49.935487 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.936315 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.937495 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:49.937859 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:49.937946 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:49.941723 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:49.941736 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:49.941752 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:49.941787 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:49.944414 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.945273 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.945284 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.945290 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.945465 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:49.945811 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:49.945814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:49.945817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:49.946183 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:49.946398 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:49.946725 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 04:45:49.946949 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:45:49.947942 8493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:50.035231 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 8371
I20260619 04:45:50.051966 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 8503
Jun 19 04:45:50 dist-test-slave-2qp8 krb5kdc[8132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:45:50.066715 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:50.052885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.3.243.131:39105)
0619 04:45:50.053083 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:50.053090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:50.053109 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:45:50.053284 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:50.053288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:50.053361 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:50.053476 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:50.054781 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:50.055503 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:50.056889 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:50.057267 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:50.057434 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:50.061540 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:50.061554 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:50.061570 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:50.061605 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:50.064435 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:50.065064 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:50.065071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:50.065073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:50.065196 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:50.065500 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:50.065503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:50.065519 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:50.065846 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:50.066058 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:50.066288 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 04:45:50.066450 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:45:50.067358 8625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:50.157999 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 8503
I20260619 04:45:50.174455 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 8145
I20260619 04:45:50.175626 8206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:50.285146 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 8145
2026-06-19T04:45:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4018 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T04:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:50Z Disabled control of system clock
I20260619 04:45:50.351542 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46813
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 04:45:50.674077 8682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:50.674382 8682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:50.674530 8682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:50.681850 8682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:50.681958 8682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:50.681999 8682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:50.682083 8682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:50.682148 8682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:50.691577 8682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:46813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:50.692981 8682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:50.694831 8682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:50.707039 8688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:50.707043 8690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:50.707041 8687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:50.708353 8682 server_base.cc:1034] running on GCE node
I20260619 04:45:50.709237 8682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:50.710660 8682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:50.711889 8682 hybrid_clock.cc:648] HybridClock initialized: now 1781844350711836 us; error 35 us; skew 500 ppm
I20260619 04:45:50.712323 8682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:45:50.714525 8682 webserver.cc:460] Webserver started at http://127.3.243.190:33243/ using document root <none> and password file <none>
I20260619 04:45:50.715221 8682 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:50.715332 8682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:50.715699 8682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:50.718644 8682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "7b9539a5d6ed473190eb8d1b0977fc44"
format_stamp: "Formatted at 2026-06-19 04:45:50 on dist-test-slave-2qp8"
server_key: "dda5dac20ab93e9039a5193360705b4a"
server_key_iv: "70bf19b53c29e06fffe4d776f0fc2380"
server_key_version: "encryptionkey@0"
I20260619 04:45:50.719521 8682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "7b9539a5d6ed473190eb8d1b0977fc44"
format_stamp: "Formatted at 2026-06-19 04:45:50 on dist-test-slave-2qp8"
server_key: "dda5dac20ab93e9039a5193360705b4a"
server_key_iv: "70bf19b53c29e06fffe4d776f0fc2380"
server_key_version: "encryptionkey@0"
I20260619 04:45:50.724525 8682 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 04:45:50.727716 8696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:50.729074 8682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:50.729241 8682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "7b9539a5d6ed473190eb8d1b0977fc44"
format_stamp: "Formatted at 2026-06-19 04:45:50 on dist-test-slave-2qp8"
server_key: "dda5dac20ab93e9039a5193360705b4a"
server_key_iv: "70bf19b53c29e06fffe4d776f0fc2380"
server_key_version: "encryptionkey@0"
I20260619 04:45:50.729444 8682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:50.735257 8682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:50.735972 8682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:50.736320 8682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:50.753942 8682 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:46813
I20260619 04:45:50.753975 8747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:46813 every 8 connection(s)
I20260619 04:45:50.755286 8682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:50.758802 8748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:50.759730 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 8682
I20260619 04:45:50.759929 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:50.760618 4046 external_mini_cluster.cc:1428] Setting key f78ff0e8209314ba138f33194a5a7160
I20260619 04:45:50.768760 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:50.762636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48222 (local address 127.3.243.190:46813)
0619 04:45:50.763621 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:50.763631 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:50.763669 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:45:50.765231 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:50.765250 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:50.765659 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:50.765970 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:50.767097 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:50.767104 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:50.767116 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:50.767129 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:50.767183 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:50.767199 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:50.767361 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:50.767504 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 04:45:50.767755 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":362,"threads_started":1}
I20260619 04:45:50.772082 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44: Bootstrap starting.
I20260619 04:45:50.774950 8748 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:50.775923 8748 log.cc:824] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:50.778816 8748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44: No bootstrap required, opened a new log
I20260619 04:45:50.785869 8748 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9539a5d6ed473190eb8d1b0977fc44" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46813 } }
I20260619 04:45:50.786249 8748 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:50.786355 8748 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9539a5d6ed473190eb8d1b0977fc44, State: Initialized, Role: FOLLOWER
I20260619 04:45:50.786908 8748 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9539a5d6ed473190eb8d1b0977fc44" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46813 } }
I20260619 04:45:50.787153 8748 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:50.787295 8748 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:50.787453 8748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:50.789633 8748 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9539a5d6ed473190eb8d1b0977fc44" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46813 } }
I20260619 04:45:50.790100 8748 leader_election.cc:302] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b9539a5d6ed473190eb8d1b0977fc44; no voters:
I20260619 04:45:50.790967 8748 leader_election.cc:288] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:50.791095 8753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:50.792358 8753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [term 1 LEADER]: Becoming Leader. State: Replica: 7b9539a5d6ed473190eb8d1b0977fc44, State: Running, Role: LEADER
I20260619 04:45:50.792855 8753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9539a5d6ed473190eb8d1b0977fc44" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46813 } }
I20260619 04:45:50.793176 8748 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:50.795534 8754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b9539a5d6ed473190eb8d1b0977fc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9539a5d6ed473190eb8d1b0977fc44" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46813 } } }
I20260619 04:45:50.795715 8755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b9539a5d6ed473190eb8d1b0977fc44. Latest consensus state: current_term: 1 leader_uuid: "7b9539a5d6ed473190eb8d1b0977fc44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9539a5d6ed473190eb8d1b0977fc44" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46813 } } }
I20260619 04:45:50.795989 8754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:50.795997 8755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:50.798252 8761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:50.802577 8761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:50.810384 8761 catalog_manager.cc:1347] Generated new cluster ID: 8644ffd4c74349ae8229d82b1e5de481
I20260619 04:45:50.810498 8761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:50.825943 8761 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:50.826423 8761 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:50.834496 8761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44: Generated new TSK 0
I20260619 04:45:50.835427 8761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:45:50.842303 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46813
--builtin_ntp_servers=127.3.243.148:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 04:45:51.164927 8772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:51.165205 8772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:51.165370 8772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:51.172578 8772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:51.172676 8772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:51.172891 8772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:51.181980 8772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:51.183298 8772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:51.185106 8772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:51.198365 8778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:51.198458 8780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:51.198972 8777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:51.199499 8772 server_base.cc:1034] running on GCE node
I20260619 04:45:51.200099 8772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:51.201152 8772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:51.202382 8772 hybrid_clock.cc:648] HybridClock initialized: now 1781844351202322 us; error 58 us; skew 500 ppm
I20260619 04:45:51.202705 8772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:45:51.204980 8772 webserver.cc:460] Webserver started at http://127.3.243.129:44207/ using document root <none> and password file <none>
I20260619 04:45:51.205730 8772 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:51.205861 8772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:51.206185 8772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:51.209095 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "871b2dcf93db425ca457ee2db5965187"
format_stamp: "Formatted at 2026-06-19 04:45:51 on dist-test-slave-2qp8"
server_key: "97e38aaa35db129bb1c0854df9d639ef"
server_key_iv: "639b00c77f3b6e0ca0e1b3065a8855ae"
server_key_version: "encryptionkey@0"
I20260619 04:45:51.209870 8772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "871b2dcf93db425ca457ee2db5965187"
format_stamp: "Formatted at 2026-06-19 04:45:51 on dist-test-slave-2qp8"
server_key: "97e38aaa35db129bb1c0854df9d639ef"
server_key_iv: "639b00c77f3b6e0ca0e1b3065a8855ae"
server_key_version: "encryptionkey@0"
I20260619 04:45:51.214999 8772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:45:51.218501 8786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:51.219781 8772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 04:45:51.219964 8772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "871b2dcf93db425ca457ee2db5965187"
format_stamp: "Formatted at 2026-06-19 04:45:51 on dist-test-slave-2qp8"
server_key: "97e38aaa35db129bb1c0854df9d639ef"
server_key_iv: "639b00c77f3b6e0ca0e1b3065a8855ae"
server_key_version: "encryptionkey@0"
I20260619 04:45:51.220257 8772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:51.225816 8772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:51.226586 8772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:51.226871 8772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:51.227766 8772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:51.229176 8772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:51.229295 8772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:51.229429 8772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:51.229517 8772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:51.261652 8772 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:34991
I20260619 04:45:51.261699 8898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:34991 every 8 connection(s)
I20260619 04:45:51.262970 8772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:51.265126 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 8772
I20260619 04:45:51.265303 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:51.266080 4046 external_mini_cluster.cc:1428] Setting key bdc9a0801ff138b19beaaf67d3fc13c5
I20260619 04:45:51.268783 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46813
--builtin_ntp_servers=127.3.243.148:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 04:45:51.271617 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:51.265694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:46331 (local address 127.3.243.190:46813)
0619 04:45:51.265923 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:51.265931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:51.267481 (+ 1550us) server_negotiation.cc:407] Connection header received
0619 04:45:51.269405 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:51.269411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:51.269492 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:51.269598 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:51.270900 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:51.270905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:51.270908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:51.270923 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:51.270955 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:51.270968 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:51.271087 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:51.271279 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 04:45:51.271348 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 04:45:51.272614 8901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:51.266270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:46813 (local address 127.3.243.129:46331)
0619 04:45:51.267331 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:51.267377 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:51.269144 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:51.269826 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:51.269834 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:51.269926 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:51.270606 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:45:51.270609 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:51.270645 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:45:51.270727 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:51.271060 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:51.271071 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:51.271078 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 04:45:51.271360 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 04:45:51.271618 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":330,"threads_started":1}
I20260619 04:45:51.274185 8899 heartbeater.cc:347] Connected to a master server at 127.3.243.190:46813
I20260619 04:45:51.274516 8899 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:51.275166 8899 heartbeater.cc:511] Master 127.3.243.190:46813 requested a full tablet report, sending...
I20260619 04:45:51.277390 8713 ts_manager.cc:194] Registered new tserver with Master: 871b2dcf93db425ca457ee2db5965187 (127.3.243.129:34991)
W20260619 04:45:51.591521 8902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:51.591840 8902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:51.592001 8902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:51.599838 8902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:51.599959 8902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:51.600169 8902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:51.610232 8902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:51.611629 8902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:51.613634 8902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:51.627965 8908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:51.628028 8907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:51.627894 8910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:51.629528 8902 server_base.cc:1034] running on GCE node
I20260619 04:45:51.630308 8902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:51.631439 8902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:51.632733 8902 hybrid_clock.cc:648] HybridClock initialized: now 1781844351632643 us; error 59 us; skew 500 ppm
I20260619 04:45:51.633167 8902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:45:51.635713 8902 webserver.cc:460] Webserver started at http://127.3.243.130:46069/ using document root <none> and password file <none>
I20260619 04:45:51.636528 8902 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:51.636633 8902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:51.636911 8902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:51.640223 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b"
format_stamp: "Formatted at 2026-06-19 04:45:51 on dist-test-slave-2qp8"
server_key: "8559fbb80ca04c876b3ad7d64fe17b51"
server_key_iv: "56abeec1f6ca45743207e8e49c684d85"
server_key_version: "encryptionkey@0"
I20260619 04:45:51.641081 8902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b"
format_stamp: "Formatted at 2026-06-19 04:45:51 on dist-test-slave-2qp8"
server_key: "8559fbb80ca04c876b3ad7d64fe17b51"
server_key_iv: "56abeec1f6ca45743207e8e49c684d85"
server_key_version: "encryptionkey@0"
I20260619 04:45:51.646683 8902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:45:51.650605 8916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:51.652354 8902 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:51.652588 8902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b"
format_stamp: "Formatted at 2026-06-19 04:45:51 on dist-test-slave-2qp8"
server_key: "8559fbb80ca04c876b3ad7d64fe17b51"
server_key_iv: "56abeec1f6ca45743207e8e49c684d85"
server_key_version: "encryptionkey@0"
I20260619 04:45:51.652915 8902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:51.659705 8902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:51.660717 8902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:51.661026 8902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:51.662065 8902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:51.663582 8902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:51.663700 8902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:51.663846 8902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:51.663942 8902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:51.698602 8902 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:39885
I20260619 04:45:51.698671 9028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:39885 every 8 connection(s)
I20260619 04:45:51.699988 8902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:51.708077 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:51.702604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:54347 (local address 127.3.243.190:46813)
0619 04:45:51.702805 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:51.702810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:51.704452 (+ 1642us) server_negotiation.cc:407] Connection header received
0619 04:45:51.706197 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:51.706201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:51.706253 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:51.706339 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:51.707524 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:51.707529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:51.707532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:51.707546 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:51.707574 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:51.707585 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:51.707672 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:51.707826 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 04:45:51.707872 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:45:51.709026 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:51.703272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:46813 (local address 127.3.243.130:54347)
0619 04:45:51.704292 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:51.704333 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:51.705856 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:51.706563 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:51.706571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:51.706655 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:51.707242 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:45:51.707244 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:51.707279 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:45:51.707360 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:51.707664 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:51.707670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:51.707675 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:45:51.707865 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 04:45:51.708046 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":281,"threads_started":1}
I20260619 04:45:51.710103 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 8902
I20260619 04:45:51.710268 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:51.710376 9029 heartbeater.cc:347] Connected to a master server at 127.3.243.190:46813
I20260619 04:45:51.710680 9029 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:51.710810 4046 external_mini_cluster.cc:1428] Setting key af73d192268a66ad4110fdfc65cb517b
I20260619 04:45:51.711299 9029 heartbeater.cc:511] Master 127.3.243.190:46813 requested a full tablet report, sending...
I20260619 04:45:51.712694 8713 ts_manager.cc:194] Registered new tserver with Master: cae3b075a4ac4ad0a0fd761744fb0b8b (127.3.243.130:39885)
I20260619 04:45:51.713701 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46813
--builtin_ntp_servers=127.3.243.148:33913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 04:45:52.032624 9032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:52.032963 9032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:52.033080 9032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:52.040613 9032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:52.040725 9032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:52.040838 9032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:45:52.050005 9032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:33913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:52.051357 9032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:52.053072 9032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:52.067021 9040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:52.067018 9037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:52.067481 9038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:52.067854 9032 server_base.cc:1034] running on GCE node
I20260619 04:45:52.068543 9032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:52.069473 9032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:52.070727 9032 hybrid_clock.cc:648] HybridClock initialized: now 1781844352070672 us; error 40 us; skew 500 ppm
I20260619 04:45:52.071049 9032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:45:52.073326 9032 webserver.cc:460] Webserver started at http://127.3.243.131:46483/ using document root <none> and password file <none>
I20260619 04:45:52.074020 9032 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:52.074150 9032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:52.074461 9032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:52.077343 9032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "3072d804262f40ed91ed9a5cfa277c3a"
format_stamp: "Formatted at 2026-06-19 04:45:52 on dist-test-slave-2qp8"
server_key: "8f3e979104bf778075dacebb2816411b"
server_key_iv: "5ab892dee0ca2d9276caca597d3de802"
server_key_version: "encryptionkey@0"
I20260619 04:45:52.078130 9032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "3072d804262f40ed91ed9a5cfa277c3a"
format_stamp: "Formatted at 2026-06-19 04:45:52 on dist-test-slave-2qp8"
server_key: "8f3e979104bf778075dacebb2816411b"
server_key_iv: "5ab892dee0ca2d9276caca597d3de802"
server_key_version: "encryptionkey@0"
I20260619 04:45:52.083117 9032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260619 04:45:52.086428 9046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:52.087616 9032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 04:45:52.087785 9032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "3072d804262f40ed91ed9a5cfa277c3a"
format_stamp: "Formatted at 2026-06-19 04:45:52 on dist-test-slave-2qp8"
server_key: "8f3e979104bf778075dacebb2816411b"
server_key_iv: "5ab892dee0ca2d9276caca597d3de802"
server_key_version: "encryptionkey@0"
I20260619 04:45:52.088024 9032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:52.093614 9032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:52.094374 9032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:52.094676 9032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:52.095583 9032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:52.096999 9032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:52.097122 9032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:52.097244 9032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:52.097329 9032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:52.129858 9032 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:37867
I20260619 04:45:52.129890 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:37867 every 8 connection(s)
I20260619 04:45:52.131388 9032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:45:52.131986 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 9032
I20260619 04:45:52.132206 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:45:52.132819 4046 external_mini_cluster.cc:1428] Setting key a514bdbb2e955daa5ff0e491023c6b31
I20260619 04:45:52.140573 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.134674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:44161 (local address 127.3.243.190:46813)
0619 04:45:52.134911 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.134917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.136679 (+ 1762us) server_negotiation.cc:407] Connection header received
0619 04:45:52.138322 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.138327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.138391 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.138519 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.139822 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.139828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.139833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.139851 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.139893 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.139904 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.140031 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.140239 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.140306 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:45:52.141444 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.135431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:46813 (local address 127.3.243.131:44161)
0619 04:45:52.136535 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.136571 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:52.138067 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:52.138732 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:52.138740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:52.138829 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:52.139533 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:45:52.139536 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:52.139573 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:45:52.139653 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:52.139996 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:52.140015 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:52.140022 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 04:45:52.140319 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 04:45:52.140563 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":379,"threads_started":1}
I20260619 04:45:52.142731 9159 heartbeater.cc:347] Connected to a master server at 127.3.243.190:46813
I20260619 04:45:52.142988 9159 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:52.143469 9159 heartbeater.cc:511] Master 127.3.243.190:46813 requested a full tablet report, sending...
I20260619 04:45:52.144778 8713 ts_manager.cc:194] Registered new tserver with Master: 3072d804262f40ed91ed9a5cfa277c3a (127.3.243.131:37867)
I20260619 04:45:52.148285 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:45:52.154924 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.152561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.3.243.190:46813)
0619 04:45:52.152718 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.152722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.153257 (+ 535us) server_negotiation.cc:407] Connection header received
0619 04:45:52.153367 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.153371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.153417 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.153497 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.154386 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.154390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.154392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.154404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.154428 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.154438 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.154525 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.154589 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.154630 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:45:52.158962 8713 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:45:52.165601 8713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:45:52.194231 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.176921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34991 (local address 127.0.0.1:44458)
0619 04:45:52.189381 (+ 12460us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.189433 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:52.189564 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:52.191683 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:52.191693 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:52.191789 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:52.192660 (+ 871us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:45:52.192663 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:52.192694 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:45:52.192809 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:52.193355 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:52.193370 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:52.193378 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 04:45:52.193573 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 04:45:52.193662 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12197,"thread_start_us":14437,"threads_started":1}
I20260619 04:45:52.195432 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.177112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44458 (local address 127.3.243.129:34991)
0619 04:45:52.189812 (+ 12700us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.189822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.189854 (+ 32us) server_negotiation.cc:407] Connection header received
0619 04:45:52.189930 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.189939 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.190221 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.190393 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.193027 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.193042 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.193064 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.193091 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.193171 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.193198 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.194877 (+ 1679us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.194960 (+ 83us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.195082 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12524,"thread_start_us":13438,"threads_started":1}
I20260619 04:45:52.196218 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.177371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.3.243.130:39885)
0619 04:45:52.188614 (+ 11243us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.188625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.191471 (+ 2846us) server_negotiation.cc:407] Connection header received
0619 04:45:52.191552 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.191563 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.191792 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.191942 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.194618 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.194631 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.194647 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.194677 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.194740 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.194765 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.194937 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.195739 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.195843 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11005,"thread_start_us":11247,"threads_started":1}
I20260619 04:45:52.196713 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.190767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.3.243.131:37867)
0619 04:45:52.193137 (+ 2370us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.193147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.193683 (+ 536us) server_negotiation.cc:407] Connection header received
0619 04:45:52.193943 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.193949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.194181 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.194305 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.195890 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.195900 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.195915 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.195929 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.195971 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.195986 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.196195 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.196257 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.196340 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2209,"thread_start_us":1557,"threads_started":1}
I20260619 04:45:52.196732 9171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.177202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:39885 (local address 127.0.0.1:53964)
0619 04:45:52.191129 (+ 13927us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.191148 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:52.191253 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:52.193077 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:52.193081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:52.193104 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:52.194220 (+ 1116us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:45:52.194226 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:52.194246 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:45:52.194290 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:52.195006 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:52.195011 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:52.195016 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:45:52.195183 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 04:45:52.195244 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13783,"thread_start_us":12356,"threads_started":1}
I20260619 04:45:52.197180 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.190400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:37867 (local address 127.0.0.1:35426)
0619 04:45:52.193581 (+ 3181us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.193598 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:52.193770 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:52.194972 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:52.194976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:52.194996 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:52.195694 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:45:52.195696 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:52.195708 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:45:52.195740 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:52.196092 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:52.196096 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:52.196101 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:45:52.196227 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 04:45:52.196274 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3062,"thread_start_us":2900,"threads_started":1}
I20260619 04:45:52.199541 8834 tablet_service.cc:1467] Processing CreateTablet for tablet 919408b8055d4c5f9c96e6f46b9fb786 (DEFAULT_TABLE table=test-table [id=06fc29a3708b43509b74380b3499cd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:52.200165 9094 tablet_service.cc:1467] Processing CreateTablet for tablet 919408b8055d4c5f9c96e6f46b9fb786 (DEFAULT_TABLE table=test-table [id=06fc29a3708b43509b74380b3499cd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:52.200392 8964 tablet_service.cc:1467] Processing CreateTablet for tablet 919408b8055d4c5f9c96e6f46b9fb786 (DEFAULT_TABLE table=test-table [id=06fc29a3708b43509b74380b3499cd5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:45:52.201063 8834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919408b8055d4c5f9c96e6f46b9fb786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:52.201529 9094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919408b8055d4c5f9c96e6f46b9fb786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:52.201714 8964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919408b8055d4c5f9c96e6f46b9fb786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:52.212432 9177 tablet_bootstrap.cc:492] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: Bootstrap starting.
I20260619 04:45:52.215637 9178 tablet_bootstrap.cc:492] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187: Bootstrap starting.
I20260619 04:45:52.217264 9177 tablet_bootstrap.cc:654] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:52.217344 9179 tablet_bootstrap.cc:492] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: Bootstrap starting.
I20260619 04:45:52.218636 9177 log.cc:824] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:52.219129 9178 tablet_bootstrap.cc:654] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:52.220249 9179 tablet_bootstrap.cc:654] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:52.220211 9178 log.cc:824] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:52.221360 9179 log.cc:824] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:52.221743 9177 tablet_bootstrap.cc:492] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: No bootstrap required, opened a new log
I20260619 04:45:52.222008 9177 ts_tablet_manager.cc:1397] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 04:45:52.223080 9178 tablet_bootstrap.cc:492] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187: No bootstrap required, opened a new log
I20260619 04:45:52.223402 9178 ts_tablet_manager.cc:1397] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260619 04:45:52.223982 9179 tablet_bootstrap.cc:492] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: No bootstrap required, opened a new log
I20260619 04:45:52.224367 9179 ts_tablet_manager.cc:1397] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 04:45:52.231343 9179 raft_consensus.cc:348] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } }
I20260619 04:45:52.231711 9179 raft_consensus.cc:374] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:52.231802 9179 raft_consensus.cc:729] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae3b075a4ac4ad0a0fd761744fb0b8b, State: Initialized, Role: FOLLOWER
I20260619 04:45:52.232415 9179 consensus_queue.cc:260] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } }
I20260619 04:45:52.233608 9029 heartbeater.cc:503] Master 127.3.243.190:46813 was elected leader, sending a full tablet report...
I20260619 04:45:52.233652 9178 raft_consensus.cc:348] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } }
I20260619 04:45:52.234079 9178 raft_consensus.cc:374] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:52.234185 9178 raft_consensus.cc:729] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871b2dcf93db425ca457ee2db5965187, State: Initialized, Role: FOLLOWER
I20260619 04:45:52.234299 9179 ts_tablet_manager.cc:1428] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20260619 04:45:52.233175 9177 raft_consensus.cc:348] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } }
I20260619 04:45:52.234763 9178 consensus_queue.cc:260] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } }
I20260619 04:45:52.235055 9177 raft_consensus.cc:374] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:52.235164 9177 raft_consensus.cc:729] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3072d804262f40ed91ed9a5cfa277c3a, State: Initialized, Role: FOLLOWER
I20260619 04:45:52.235806 8899 heartbeater.cc:503] Master 127.3.243.190:46813 was elected leader, sending a full tablet report...
I20260619 04:45:52.236604 9178 ts_tablet_manager.cc:1428] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 04:45:52.236756 9177 consensus_queue.cc:260] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } }
I20260619 04:45:52.238411 9159 heartbeater.cc:503] Master 127.3.243.190:46813 was elected leader, sending a full tablet report...
I20260619 04:45:52.239081 9177 ts_tablet_manager.cc:1428] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260619 04:45:52.265125 8900 tablet.cc:2369] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:45:52.305758 9184 raft_consensus.cc:482] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:45:52.306033 9184 raft_consensus.cc:504] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } }
I20260619 04:45:52.307686 9184 leader_election.cc:288] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae3b075a4ac4ad0a0fd761744fb0b8b (127.3.243.130:39885), 3072d804262f40ed91ed9a5cfa277c3a (127.3.243.131:37867)
I20260619 04:45:52.311111 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.308127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:37091 (local address 127.3.243.130:39885)
0619 04:45:52.308307 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.308311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.308918 (+ 607us) server_negotiation.cc:407] Connection header received
0619 04:45:52.309105 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.309114 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.309199 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.309303 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.310488 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.310492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.310495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.310511 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.310542 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.310553 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.310656 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.310782 (+ 126us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.310846 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:45:52.311321 9186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.308102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:39885 (local address 127.3.243.129:37091)
0619 04:45:52.308764 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.308780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:52.308946 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:52.309390 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:52.309396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:52.309420 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:52.310252 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:45:52.310256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:52.310279 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:45:52.310332 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:52.310646 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:52.310649 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:52.310653 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 04:45:52.310797 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 04:45:52.310845 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":336,"threads_started":1}
I20260619 04:45:52.311798 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.309085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:55589 (local address 127.3.243.131:37867)
0619 04:45:52.309278 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.309284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.309627 (+ 343us) server_negotiation.cc:407] Connection header received
0619 04:45:52.309755 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.309777 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.309856 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.309950 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.311233 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.311243 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.311246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.311261 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.311292 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.311303 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.311427 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.311567 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.311629 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 04:45:52.312414 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.308920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:37867 (local address 127.3.243.129:55589)
0619 04:45:52.309505 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.309526 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:52.309637 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:52.309978 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:52.310002 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:52.310045 (+ 43us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:52.311033 (+ 988us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:45:52.311036 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:52.311048 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:45:52.311080 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:52.311405 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:52.311410 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:52.311416 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:45:52.312034 (+ 618us) client_negotiation.cc:240] Negotiation successful
0619 04:45:52.312074 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":262,"threads_started":1}
I20260619 04:45:52.312317 8984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919408b8055d4c5f9c96e6f46b9fb786" candidate_uuid: "871b2dcf93db425ca457ee2db5965187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" is_pre_election: true
I20260619 04:45:52.312871 8984 raft_consensus.cc:2454] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 871b2dcf93db425ca457ee2db5965187 in term 0.
I20260619 04:45:52.313571 8789 leader_election.cc:302] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 871b2dcf93db425ca457ee2db5965187, cae3b075a4ac4ad0a0fd761744fb0b8b; no voters:
I20260619 04:45:52.313493 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919408b8055d4c5f9c96e6f46b9fb786" candidate_uuid: "871b2dcf93db425ca457ee2db5965187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3072d804262f40ed91ed9a5cfa277c3a" is_pre_election: true
I20260619 04:45:52.313957 9184 raft_consensus.cc:2781] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:45:52.313973 9114 raft_consensus.cc:2454] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 871b2dcf93db425ca457ee2db5965187 in term 0.
I20260619 04:45:52.314124 9184 raft_consensus.cc:482] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:45:52.314240 9184 raft_consensus.cc:3037] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:52.316623 9184 raft_consensus.cc:504] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } }
I20260619 04:45:52.317251 9184 leader_election.cc:288] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [CANDIDATE]: Term 1 election: Requested vote from peers cae3b075a4ac4ad0a0fd761744fb0b8b (127.3.243.130:39885), 3072d804262f40ed91ed9a5cfa277c3a (127.3.243.131:37867)
I20260619 04:45:52.317657 8984 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919408b8055d4c5f9c96e6f46b9fb786" candidate_uuid: "871b2dcf93db425ca457ee2db5965187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b"
I20260619 04:45:52.317760 9114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919408b8055d4c5f9c96e6f46b9fb786" candidate_uuid: "871b2dcf93db425ca457ee2db5965187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3072d804262f40ed91ed9a5cfa277c3a"
I20260619 04:45:52.317857 8984 raft_consensus.cc:3037] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:52.317994 9114 raft_consensus.cc:3037] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:52.320212 8984 raft_consensus.cc:2454] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 871b2dcf93db425ca457ee2db5965187 in term 1.
I20260619 04:45:52.320246 9114 raft_consensus.cc:2454] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 871b2dcf93db425ca457ee2db5965187 in term 1.
I20260619 04:45:52.320614 8789 leader_election.cc:302] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3072d804262f40ed91ed9a5cfa277c3a, 871b2dcf93db425ca457ee2db5965187; no voters:
I20260619 04:45:52.320904 9184 raft_consensus.cc:2781] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:52.321341 9184 raft_consensus.cc:686] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 1 LEADER]: Becoming Leader. State: Replica: 871b2dcf93db425ca457ee2db5965187, State: Running, Role: LEADER
I20260619 04:45:52.321924 9184 consensus_queue.cc:237] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } }
I20260619 04:45:52.326367 8713 catalog_manager.cc:5495] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 reported cstate change: term changed from 0 to 1, leader changed from <none> to 871b2dcf93db425ca457ee2db5965187 (127.3.243.129). New cstate: current_term: 1 leader_uuid: "871b2dcf93db425ca457ee2db5965187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871b2dcf93db425ca457ee2db5965187" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 } health_report { overall_health: UNKNOWN } } }
I20260619 04:45:52.338948 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.336992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44474 (local address 127.3.243.129:34991)
0619 04:45:52.337190 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.337195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.337298 (+ 103us) server_negotiation.cc:407] Connection header received
0619 04:45:52.337459 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.337462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.337539 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.337619 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.338366 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.338370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.338373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.338387 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.338416 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.338428 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.338515 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.338608 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.338652 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:45:52.346730 8984 raft_consensus.cc:1261] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [term 1 FOLLOWER]: Refusing update from remote peer 871b2dcf93db425ca457ee2db5965187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:52.346730 9114 raft_consensus.cc:1261] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [term 1 FOLLOWER]: Refusing update from remote peer 871b2dcf93db425ca457ee2db5965187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:45:52.347661 9188 consensus_queue.cc:1035] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3072d804262f40ed91ed9a5cfa277c3a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 37867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:52.348012 9184 consensus_queue.cc:1035] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae3b075a4ac4ad0a0fd761744fb0b8b" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:45:52.373163 8713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:52.373442 8713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:45:52.376781 8713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b9539a5d6ed473190eb8d1b0977fc44: Sending DeleteTablet for 3 replicas of tablet 919408b8055d4c5f9c96e6f46b9fb786
I20260619 04:45:52.377914 8834 tablet_service.cc:1514] Processing DeleteTablet for tablet 919408b8055d4c5f9c96e6f46b9fb786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:52 UTC) from {username='slave'} at 127.0.0.1:44458
I20260619 04:45:52.378096 8964 tablet_service.cc:1514] Processing DeleteTablet for tablet 919408b8055d4c5f9c96e6f46b9fb786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:52 UTC) from {username='slave'} at 127.0.0.1:53964
I20260619 04:45:52.378320 9094 tablet_service.cc:1514] Processing DeleteTablet for tablet 919408b8055d4c5f9c96e6f46b9fb786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:45:52 UTC) from {username='slave'} at 127.0.0.1:35426
I20260619 04:45:52.379552 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 8772
I20260619 04:45:52.391273 9202 tablet_replica.cc:330] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: stopping tablet replica
I20260619 04:45:52.391891 9202 raft_consensus.cc:2229] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:52.392640 9202 raft_consensus.cc:2258] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:52.393656 9201 tablet_replica.cc:330] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187: stopping tablet replica
I20260619 04:45:52.394469 9203 tablet_replica.cc:330] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: stopping tablet replica
I20260619 04:45:52.395079 9203 raft_consensus.cc:2229] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:45:52.395602 9203 raft_consensus.cc:2258] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:45:52.395673 9202 ts_tablet_manager.cc:1905] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:52.394906 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.381225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.3.243.129:34991)
0619 04:45:52.381437 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.381446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.390924 (+ 9478us) server_negotiation.cc:407] Connection header received
0619 04:45:52.391457 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.391465 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.391541 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.391632 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.393856 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.393864 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.393870 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.393891 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.393934 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.393950 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.394076 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.394559 (+ 483us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.394622 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:45:52.396893 8894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:52.402390 9202 ts_tablet_manager.cc:1918] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:45:52.402570 9202 log.cc:1196] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/919408b8055d4c5f9c96e6f46b9fb786
I20260619 04:45:52.403049 9202 ts_tablet_manager.cc:1939] T 919408b8055d4c5f9c96e6f46b9fb786 P cae3b075a4ac4ad0a0fd761744fb0b8b: Deleting consensus metadata
I20260619 04:45:52.403522 9203 ts_tablet_manager.cc:1905] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:45:52.404894 8699 catalog_manager.cc:4841] TS cae3b075a4ac4ad0a0fd761744fb0b8b (127.3.243.130:39885): tablet 919408b8055d4c5f9c96e6f46b9fb786 (table test-table [id=06fc29a3708b43509b74380b3499cd5a]) successfully deleted
I20260619 04:45:52.409200 9203 ts_tablet_manager.cc:1918] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:45:52.409389 9203 log.cc:1196] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/919408b8055d4c5f9c96e6f46b9fb786
I20260619 04:45:52.409924 9203 ts_tablet_manager.cc:1939] T 919408b8055d4c5f9c96e6f46b9fb786 P 3072d804262f40ed91ed9a5cfa277c3a: Deleting consensus metadata
I20260619 04:45:52.411561 8699 catalog_manager.cc:4841] TS 3072d804262f40ed91ed9a5cfa277c3a (127.3.243.131:37867): tablet 919408b8055d4c5f9c96e6f46b9fb786 (table test-table [id=06fc29a3708b43509b74380b3499cd5a]) successfully deleted
I20260619 04:45:52.397078 9201 raft_consensus.cc:2229] T 919408b8055d4c5f9c96e6f46b9fb786 P 871b2dcf93db425ca457ee2db5965187 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:45:52.491487 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 8772
I20260619 04:45:52.509042 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 8902
W20260619 04:45:52.509014 8698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.129:34991 (error 108)
I20260619 04:45:52.511032 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.510513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34991 (local address 127.0.0.1:44498)
0619 04:45:52.510714 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.510828 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 04:45:52.511370 8698 catalog_manager.cc:4584] TS 871b2dcf93db425ca457ee2db5965187 (127.3.243.129:34991): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 919408b8055d4c5f9c96e6f46b9fb786: Network error: Client connection negotiation failed: client connection to 127.3.243.129:34991: connect: Connection refused (error 111)
I20260619 04:45:52.512329 9174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.509899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.3.243.130:39885)
0619 04:45:52.510104 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.510112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.510135 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:45:52.510307 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.510313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.510421 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.510515 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.511635 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.511644 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.511650 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.511668 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.511707 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.511721 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.511834 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.511954 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.512003 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:45:52.512992 9024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:52.565373 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.564855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34991 (local address 127.0.0.1:44506)
0619 04:45:52.565102 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.565199 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 04:45:52.566421 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.566086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34991 (local address 127.0.0.1:44514)
0619 04:45:52.566232 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.566317 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:45:52.600430 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 8902
I20260619 04:45:52.617753 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 9032
I20260619 04:45:52.620283 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.618556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.3.243.131:37867)
0619 04:45:52.618716 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:52.618721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:52.618737 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:45:52.618791 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:52.618795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:52.618866 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:52.618949 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:52.619729 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:52.619734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:52.619737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:52.619752 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:52.619781 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:52.619793 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:52.619893 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:52.619949 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 04:45:52.619992 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:45:52.620939 9154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:52.626709 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.626169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34991 (local address 127.0.0.1:44520)
0619 04:45:52.626379 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.626529 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 04:45:52.627799 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.627396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34991 (local address 127.0.0.1:44536)
0619 04:45:52.627559 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.627650 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 04:45:52.708549 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.707986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34991 (local address 127.0.0.1:44538)
0619 04:45:52.708249 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.708376 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 04:45:52.709273 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 9032
I20260619 04:45:52.710100 9175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:52.709549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34991 (local address 127.0.0.1:44542)
0619 04:45:52.709900 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:52.709986 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:34991: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 04:45:52.726164 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 8682
I20260619 04:45:52.727365 8743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:52.833175 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 8682
2026-06-19T04:45:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2543 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 04:45:52.870270 4046 mini_oidc.cc:152] Starting JWKS server
I20260619 04:45:52.872830 4046 webserver.cc:460] Webserver started at https://127.0.0.1:38321/ using document root <none> and password file <none>
I20260619 04:45:52.873092 4046 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 04:45:52.873682 4046 webserver.cc:460] Webserver started at http://127.0.0.1:36509/ using document root <none> and password file <none>
2026-06-19T04:45:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:52Z Disabled control of system clock
I20260619 04:45:52.900069 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:37327
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:41107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:37327
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38321/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/testchainca.cert with env {}
W20260619 04:45:53.220007 9215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:53.220381 9215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:53.220472 9215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:53.227883 9215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:53.228005 9215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:53.228044 9215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:53.228093 9215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:53.228154 9215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 04:45:53.228215 9215 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 04:45:53.228255 9215 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38321/jwks.json
I20260619 04:45:53.238200 9215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:41107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:37327
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:37327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38321/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:53.239444 9215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:53.240672 9215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:53.252384 9221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:53.252326 9223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:53.252435 9220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:53.253443 9215 server_base.cc:1034] running on GCE node
I20260619 04:45:53.254243 9215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:53.255594 9215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:53.256819 9215 hybrid_clock.cc:648] HybridClock initialized: now 1781844353256784 us; error 50 us; skew 500 ppm
I20260619 04:45:53.257161 9215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:45:53.259411 9215 webserver.cc:460] Webserver started at http://127.3.243.190:46617/ using document root <none> and password file <none>
I20260619 04:45:53.260088 9215 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:53.260269 9215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:53.260577 9215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:53.263381 9215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4"
format_stamp: "Formatted at 2026-06-19 04:45:53 on dist-test-slave-2qp8"
server_key: "b9dad8adf3780b899e619ec43f96a1dc"
server_key_iv: "a060a03c5e3e9f0a3e2797343cfe2c3b"
server_key_version: "encryptionkey@0"
I20260619 04:45:53.264225 9215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4"
format_stamp: "Formatted at 2026-06-19 04:45:53 on dist-test-slave-2qp8"
server_key: "b9dad8adf3780b899e619ec43f96a1dc"
server_key_iv: "a060a03c5e3e9f0a3e2797343cfe2c3b"
server_key_version: "encryptionkey@0"
I20260619 04:45:53.268975 9215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 04:45:53.272176 9229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:53.273409 9215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:45:53.273602 9215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4"
format_stamp: "Formatted at 2026-06-19 04:45:53 on dist-test-slave-2qp8"
server_key: "b9dad8adf3780b899e619ec43f96a1dc"
server_key_iv: "a060a03c5e3e9f0a3e2797343cfe2c3b"
server_key_version: "encryptionkey@0"
I20260619 04:45:53.273801 9215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:53.294261 9215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:53.295143 9215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:53.295469 9215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:53.313891 9215 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:37327
I20260619 04:45:53.313925 9280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:37327 every 8 connection(s)
I20260619 04:45:53.315311 9215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:53.318122 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 9215
I20260619 04:45:53.318341 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:53.319015 4046 external_mini_cluster.cc:1428] Setting key 93f0f287d95221a3b44bb4ee15bc8bf6
I20260619 04:45:53.319239 9281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:53.334291 9281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4: Bootstrap starting.
I20260619 04:45:53.334532 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:53.321046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.3.243.190:37327)
0619 04:45:53.322078 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:53.322105 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:53.322156 (+ 51us) server_negotiation.cc:407] Connection header received
0619 04:45:53.324275 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:53.324303 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:53.324897 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:53.325341 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:53.325864 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:53.327821 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:53.329321 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:53.330388 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:53.330591 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:53.331697 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:53.331707 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:53.331728 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:53.332485 (+ 757us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:53.332578 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:53.332603 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:53.332708 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:53.332925 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 04:45:53.333255 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":295,"threads_started":1}
I20260619 04:45:53.337685 9281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:53.338210 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:45:53.338898 9281 log.cc:824] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:53.341796 9281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4: No bootstrap required, opened a new log
I20260619 04:45:53.348907 9281 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37327 } }
I20260619 04:45:53.349323 9281 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:53.349433 9281 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1dd6046c1e4104b8147b6ee5dbb3f4, State: Initialized, Role: FOLLOWER
I20260619 04:45:53.350018 9281 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37327 } }
I20260619 04:45:53.350257 9281 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:53.350402 9281 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:53.350584 9281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:53.352859 9281 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37327 } }
I20260619 04:45:53.353389 9281 leader_election.cc:302] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad1dd6046c1e4104b8147b6ee5dbb3f4; no voters:
I20260619 04:45:53.353948 9281 leader_election.cc:288] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:53.354115 9286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:53.354861 9286 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [term 1 LEADER]: Becoming Leader. State: Replica: ad1dd6046c1e4104b8147b6ee5dbb3f4, State: Running, Role: LEADER
I20260619 04:45:53.355403 9286 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37327 } }
I20260619 04:45:53.356154 9281 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:53.358273 9287 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37327 } } }
I20260619 04:45:53.358731 9287 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:53.361181 9288 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad1dd6046c1e4104b8147b6ee5dbb3f4. Latest consensus state: current_term: 1 leader_uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad1dd6046c1e4104b8147b6ee5dbb3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 37327 } } }
I20260619 04:45:53.361543 9288 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:53.362027 9295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:53.366473 9295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:53.374640 9295 catalog_manager.cc:1347] Generated new cluster ID: 4e1c32d423854d32b3eb3514cc32a968
I20260619 04:45:53.374779 9295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:53.393327 9295 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:53.395004 9295 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:53.407577 9295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad1dd6046c1e4104b8147b6ee5dbb3f4: Generated new TSK 0
I20260619 04:45:53.408727 9295 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 04:45:53.452142 9311 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 04:45:53.452342 9311 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 04:45:53.453477 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:53.431931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44526 (local address 127.3.243.190:37327)
0619 04:45:53.432097 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:53.432136 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:53.432567 (+ 431us) server_negotiation.cc:407] Connection header received
0619 04:45:53.432884 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:53.432891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:53.432999 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:53.433158 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0619 04:45:53.433633 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:53.434792 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:53.436157 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:53.436872 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:53.436990 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:53.437527 (+ 537us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 04:45:53.452775 (+ 15248us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 04:45:53.453149 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 04:45:53.453611 9284 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44526: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 04:45:53.454775 9310 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:45:53.431759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:37327 (local address 127.0.0.1:44526)
0619 04:45:53.432462 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:53.432478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:53.432618 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:53.433188 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:53.433192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:53.433200 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 04:45:53.433493 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:53.433499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:53.434929 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:53.434933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:53.435996 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:53.436016 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:53.437038 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:53.437042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:53.437160 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:53.437214 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 04:45:53.453094 (+ 15880us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 04:45:53.453350 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.243.190:37327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":373,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.243.190:37327: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38321/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 04:45:53.456434 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 9215
I20260619 04:45:53.457739 9276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:53.553639 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 9215
2026-06-19T04:45:53Z chronyd exiting
I20260619 04:45:53.604028 4046 test_util.cc:175] -----------------------------------------------
I20260619 04:45:53.604331 4046 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781844312210132-4046-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 04:45:53.608505 4046 mini_oidc.cc:152] Starting JWKS server
I20260619 04:45:53.610626 4046 webserver.cc:460] Webserver started at https://127.0.0.1:32861/ using document root <none> and password file <none>
I20260619 04:45:53.610991 4046 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 04:45:53.611660 4046 webserver.cc:460] Webserver started at http://127.0.0.1:45351/ using document root <none> and password file <none>
2026-06-19T04:45:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:53Z Disabled control of system clock
I20260619 04:45:53.638253 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42281
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:42281
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32861/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/testchainca.cert with env {}
W20260619 04:45:53.950145 9318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:53.950495 9318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:53.950663 9318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:53.957947 9318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:53.958091 9318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:53.958180 9318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:53.958284 9318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:53.958376 9318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 04:45:53.958464 9318 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 04:45:53.958542 9318 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:32861/jwks.json
I20260619 04:45:53.968597 9318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:42281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:32861/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:53.969957 9318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:53.971267 9318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:53.983035 9326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:53.983006 9323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:53.983110 9324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:53.985435 9318 server_base.cc:1034] running on GCE node
I20260619 04:45:53.986191 9318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:53.987602 9318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:53.988837 9318 hybrid_clock.cc:648] HybridClock initialized: now 1781844353988798 us; error 51 us; skew 500 ppm
I20260619 04:45:53.989210 9318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:45:53.991535 9318 webserver.cc:460] Webserver started at http://127.3.243.190:39911/ using document root <none> and password file <none>
I20260619 04:45:53.992300 9318 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:53.992463 9318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:53.992800 9318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:53.995639 9318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "0bc6803f92184c0db2883e31b56e74f4"
format_stamp: "Formatted at 2026-06-19 04:45:53 on dist-test-slave-2qp8"
server_key: "e1b264b6260f386a92d43eedc01c1ca6"
server_key_iv: "83c2c8b154819a2c47ac2541469e55a0"
server_key_version: "encryptionkey@0"
I20260619 04:45:53.996459 9318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "0bc6803f92184c0db2883e31b56e74f4"
format_stamp: "Formatted at 2026-06-19 04:45:53 on dist-test-slave-2qp8"
server_key: "e1b264b6260f386a92d43eedc01c1ca6"
server_key_iv: "83c2c8b154819a2c47ac2541469e55a0"
server_key_version: "encryptionkey@0"
I20260619 04:45:54.001237 9318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 04:45:54.004563 9332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:54.005796 9318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 04:45:54.005970 9318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "0bc6803f92184c0db2883e31b56e74f4"
format_stamp: "Formatted at 2026-06-19 04:45:53 on dist-test-slave-2qp8"
server_key: "e1b264b6260f386a92d43eedc01c1ca6"
server_key_iv: "83c2c8b154819a2c47ac2541469e55a0"
server_key_version: "encryptionkey@0"
I20260619 04:45:54.006181 9318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:54.026408 9318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:54.027245 9318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:54.027556 9318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:54.045989 9318 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:42281
I20260619 04:45:54.045995 9383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:42281 every 8 connection(s)
I20260619 04:45:54.047366 9318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:54.051090 9384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:54.055116 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 9318
I20260619 04:45:54.055315 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:54.055951 4046 external_mini_cluster.cc:1428] Setting key cb984e9c0c251240b8fe14c7ea36368c
I20260619 04:45:54.061499 9384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4: Bootstrap starting.
I20260619 04:45:54.064558 9384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:54.065554 9384 log.cc:824] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:54.068495 9384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4: No bootstrap required, opened a new log
I20260619 04:45:54.068928 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:54.057736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.3.243.190:42281)
0619 04:45:54.058682 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:54.058694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:54.058732 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:45:54.060443 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:54.060464 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:54.060909 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:54.061283 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:54.061810 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.063107 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.064486 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.065206 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.065406 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:54.066586 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:54.066600 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:54.066625 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:54.067193 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:54.067264 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:54.067288 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:54.067370 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:54.067624 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 04:45:54.067934 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":343,"threads_started":1}
I20260619 04:45:54.072618 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:45:54.076663 9384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc6803f92184c0db2883e31b56e74f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42281 } }
I20260619 04:45:54.077019 9384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:54.077142 9384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc6803f92184c0db2883e31b56e74f4, State: Initialized, Role: FOLLOWER
I20260619 04:45:54.077750 9384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc6803f92184c0db2883e31b56e74f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42281 } }
I20260619 04:45:54.077944 9384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:54.078055 9384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:54.078171 9384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:54.080245 9384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc6803f92184c0db2883e31b56e74f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42281 } }
I20260619 04:45:54.080670 9384 leader_election.cc:302] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bc6803f92184c0db2883e31b56e74f4; no voters:
I20260619 04:45:54.081147 9384 leader_election.cc:288] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:54.081305 9389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:54.082098 9389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [term 1 LEADER]: Becoming Leader. State: Replica: 0bc6803f92184c0db2883e31b56e74f4, State: Running, Role: LEADER
I20260619 04:45:54.082624 9389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc6803f92184c0db2883e31b56e74f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42281 } }
I20260619 04:45:54.083343 9384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:54.085382 9391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bc6803f92184c0db2883e31b56e74f4. Latest consensus state: current_term: 1 leader_uuid: "0bc6803f92184c0db2883e31b56e74f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc6803f92184c0db2883e31b56e74f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42281 } } }
I20260619 04:45:54.085407 9390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bc6803f92184c0db2883e31b56e74f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bc6803f92184c0db2883e31b56e74f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42281 } } }
I20260619 04:45:54.085892 9390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:54.085892 9391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:54.088716 9394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:54.093954 9394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:54.102272 9394 catalog_manager.cc:1347] Generated new cluster ID: 6e0c8326e62d4f41a043377f8ad82274
I20260619 04:45:54.102397 9394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:54.112774 9394 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:54.114135 9394 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:54.123803 9394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bc6803f92184c0db2883e31b56e74f4: Generated new TSK 0
I20260619 04:45:54.124647 9394 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 04:45:54.162420 9414 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 04:45:54.162561 9414 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 04:45:54.163434 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:54.136230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.3.243.190:42281)
0619 04:45:54.136466 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:54.136474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:54.136850 (+ 376us) server_negotiation.cc:407] Connection header received
0619 04:45:54.137081 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:54.137085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:54.137153 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:54.137270 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0619 04:45:54.137830 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.138553 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.140517 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.140955 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.141062 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:54.141611 (+ 549us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 04:45:54.162895 (+ 21284us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 04:45:54.163226 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":100}
W20260619 04:45:54.163601 9387 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45880: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 04:45:54.163852 9413 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:45:54.136038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42281 (local address 127.0.0.1:45880)
0619 04:45:54.136739 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:54.136765 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:54.136912 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:54.137319 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:54.137325 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:54.137339 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0619 04:45:54.137646 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:54.137658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.138686 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.138692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:54.140375 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:54.140390 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.141131 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.141138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:54.141336 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:54.141382 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 04:45:54.163156 (+ 21774us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 04:45:54.163428 (+ 272us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.243.190:42281: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:32861/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":357,"threads_started":1}
I20260619 04:45:54.165235 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 9318
I20260619 04:45:54.166388 9379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:54.263330 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 9318
2026-06-19T04:45:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (710 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 04:45:54.318753 4046 mini_oidc.cc:152] Starting JWKS server
I20260619 04:45:54.320633 4046 webserver.cc:460] Webserver started at https://127.0.0.1:41861/ using document root <none> and password file <none>
I20260619 04:45:54.320847 4046 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 04:45:54.321401 4046 webserver.cc:460] Webserver started at http://127.0.0.1:38859/ using document root <none> and password file <none>
2026-06-19T04:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:54Z Disabled control of system clock
I20260619 04:45:54.347798 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:39567
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:43275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:39567
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41861/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 04:45:54.665696 9421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:54.666040 9421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:54.666193 9421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:54.674069 9421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:54.674211 9421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:54.674278 9421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:54.674355 9421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:54.674422 9421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 04:45:54.674511 9421 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 04:45:54.674567 9421 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41861/jwks.json
I20260619 04:45:54.683724 9421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:43275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:39567
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:39567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41861/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:54.685130 9421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:54.686877 9421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:54.698750 9426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:54.698716 9427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:54.698933 9429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:54.699476 9421 server_base.cc:1034] running on GCE node
I20260619 04:45:54.700330 9421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:54.701683 9421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:54.702909 9421 hybrid_clock.cc:648] HybridClock initialized: now 1781844354702864 us; error 58 us; skew 500 ppm
I20260619 04:45:54.703256 9421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:45:54.705761 9421 webserver.cc:460] Webserver started at http://127.3.243.190:45319/ using document root <none> and password file <none>
I20260619 04:45:54.706463 9421 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:54.706598 9421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:54.706929 9421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:54.709846 9421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "9761c19063f44e628e40a78e6671ac58"
format_stamp: "Formatted at 2026-06-19 04:45:54 on dist-test-slave-2qp8"
server_key: "4e3827cccaaa50b64dc70bb1ee3ed8fd"
server_key_iv: "443c84635ba6e0f7db7759bf534c8c0c"
server_key_version: "encryptionkey@0"
I20260619 04:45:54.710645 9421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "9761c19063f44e628e40a78e6671ac58"
format_stamp: "Formatted at 2026-06-19 04:45:54 on dist-test-slave-2qp8"
server_key: "4e3827cccaaa50b64dc70bb1ee3ed8fd"
server_key_iv: "443c84635ba6e0f7db7759bf534c8c0c"
server_key_version: "encryptionkey@0"
I20260619 04:45:54.715848 9421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 04:45:54.719206 9435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:54.720590 9421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:54.720772 9421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "9761c19063f44e628e40a78e6671ac58"
format_stamp: "Formatted at 2026-06-19 04:45:54 on dist-test-slave-2qp8"
server_key: "4e3827cccaaa50b64dc70bb1ee3ed8fd"
server_key_iv: "443c84635ba6e0f7db7759bf534c8c0c"
server_key_version: "encryptionkey@0"
I20260619 04:45:54.720988 9421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:54.744019 9421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:54.745476 9421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:54.745927 9421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:54.765415 9421 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:39567
I20260619 04:45:54.765448 9486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:39567 every 8 connection(s)
I20260619 04:45:54.766916 9421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:54.770776 9487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:54.776773 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 9421
I20260619 04:45:54.777006 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:54.777849 4046 external_mini_cluster.cc:1428] Setting key 64120de6e0807a9c67ed219bc414f2d7
I20260619 04:45:54.782280 9487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58: Bootstrap starting.
I20260619 04:45:54.785410 9487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:54.786384 9487 log.cc:824] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:54.789367 9487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58: No bootstrap required, opened a new log
I20260619 04:45:54.791797 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:54.779986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41122 (local address 127.3.243.190:39567)
0619 04:45:54.781031 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:54.781042 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:54.781081 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:45:54.782786 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:54.782807 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:54.783256 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:54.783656 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:54.784258 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.785528 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.786894 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.787735 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.787908 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:54.789017 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:54.789027 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:54.789045 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:54.789764 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:54.789870 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:54.789906 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:54.790035 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:54.790232 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 04:45:54.790551 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":366,"threads_started":1}
I20260619 04:45:54.795543 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:45:54.798206 9487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761c19063f44e628e40a78e6671ac58" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39567 } }
I20260619 04:45:54.798606 9487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:54.798686 9487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9761c19063f44e628e40a78e6671ac58, State: Initialized, Role: FOLLOWER
I20260619 04:45:54.799247 9487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761c19063f44e628e40a78e6671ac58" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39567 } }
I20260619 04:45:54.799479 9487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:54.799628 9487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:54.799788 9487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:54.802078 9487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761c19063f44e628e40a78e6671ac58" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39567 } }
I20260619 04:45:54.802682 9487 leader_election.cc:302] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9761c19063f44e628e40a78e6671ac58; no voters:
I20260619 04:45:54.803356 9487 leader_election.cc:288] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:54.803499 9492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:54.804250 9492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [term 1 LEADER]: Becoming Leader. State: Replica: 9761c19063f44e628e40a78e6671ac58, State: Running, Role: LEADER
I20260619 04:45:54.804852 9492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761c19063f44e628e40a78e6671ac58" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39567 } }
I20260619 04:45:54.805637 9487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:54.807538 9493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9761c19063f44e628e40a78e6671ac58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761c19063f44e628e40a78e6671ac58" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39567 } } }
I20260619 04:45:54.807907 9493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:54.809218 9494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9761c19063f44e628e40a78e6671ac58. Latest consensus state: current_term: 1 leader_uuid: "9761c19063f44e628e40a78e6671ac58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9761c19063f44e628e40a78e6671ac58" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39567 } } }
I20260619 04:45:54.809747 9494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:54.814069 9498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:54.821429 9498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:54.832751 9498 catalog_manager.cc:1347] Generated new cluster ID: 837d21f489fb4075be4d5bee5927ef26
I20260619 04:45:54.832890 9498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:54.851855 9498 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:54.853267 9498 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:54.865197 9498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9761c19063f44e628e40a78e6671ac58: Generated new TSK 0
I20260619 04:45:54.866382 9498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:45:54.889947 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:54.881363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.3.243.190:39567)
0619 04:45:54.881597 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:54.881603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:54.881963 (+ 360us) server_negotiation.cc:407] Connection header received
0619 04:45:54.882224 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:54.882228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:54.882306 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:54.882442 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0619 04:45:54.883035 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.883771 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.885963 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.886417 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.886518 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:54.887141 (+ 623us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 04:45:54.889203 (+ 2062us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 04:45:54.889589 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":93}
W20260619 04:45:54.890146 9516 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:45:54.881161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:39567 (local address 127.0.0.1:41124)
0619 04:45:54.881834 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:54.881854 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:54.882021 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:54.882497 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:54.882510 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:54.882525 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0619 04:45:54.882872 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:54.882884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.883906 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.883912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:54.885780 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:54.885796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:54.886627 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:54.886636 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:54.886845 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:54.886893 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 04:45:54.889425 (+ 2532us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 04:45:54.889707 (+ 282us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.243.190:39567: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":341,"threads_started":1}
W20260619 04:45:54.890507 9490 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41861/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260619 04:45:54.891605 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 9421
I20260619 04:45:54.892962 9482 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 04:45:54.902642 9517 webserver.cc:533] Webserver: sslize failed:
E20260619 04:45:54.902808 9517 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 04:45:54.986016 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 9421
2026-06-19T04:45:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (722 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T04:45:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:55Z Disabled control of system clock
I20260619 04:45:55.068009 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:41171
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:41171
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/testchainca.cert with env {}
W20260619 04:45:55.399374 9522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:55.399679 9522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:55.399881 9522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:55.407179 9522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:55.407295 9522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:55.407346 9522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:55.407464 9522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:55.407546 9522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 04:45:55.407666 9522 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 04:45:55.407742 9522 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 04:45:55.418298 9522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:41171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:55.419701 9522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:55.421105 9522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:55.433488 9527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:55.433542 9528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:55.433564 9530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:55.434252 9522 server_base.cc:1034] running on GCE node
I20260619 04:45:55.435092 9522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:55.436689 9522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:55.437908 9522 hybrid_clock.cc:648] HybridClock initialized: now 1781844355437860 us; error 37 us; skew 500 ppm
I20260619 04:45:55.438308 9522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:45:55.440900 9522 webserver.cc:460] Webserver started at http://127.3.243.190:37259/ using document root <none> and password file <none>
I20260619 04:45:55.441645 9522 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:55.441777 9522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:55.442138 9522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:55.445326 9522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "3b6c34fddf2c4abaa68994665b4f22d0"
format_stamp: "Formatted at 2026-06-19 04:45:55 on dist-test-slave-2qp8"
server_key: "c86d9255d9e0042a30e8dad05b63a9d1"
server_key_iv: "9990d0ef176c7eab6efceda08c0402fc"
server_key_version: "encryptionkey@0"
I20260619 04:45:55.446213 9522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "3b6c34fddf2c4abaa68994665b4f22d0"
format_stamp: "Formatted at 2026-06-19 04:45:55 on dist-test-slave-2qp8"
server_key: "c86d9255d9e0042a30e8dad05b63a9d1"
server_key_iv: "9990d0ef176c7eab6efceda08c0402fc"
server_key_version: "encryptionkey@0"
I20260619 04:45:55.451555 9522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 04:45:55.455113 9536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:55.456485 9522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 04:45:55.456688 9522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "3b6c34fddf2c4abaa68994665b4f22d0"
format_stamp: "Formatted at 2026-06-19 04:45:55 on dist-test-slave-2qp8"
server_key: "c86d9255d9e0042a30e8dad05b63a9d1"
server_key_iv: "9990d0ef176c7eab6efceda08c0402fc"
server_key_version: "encryptionkey@0"
I20260619 04:45:55.456938 9522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:55.477226 9522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:55.478142 9522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:55.478443 9522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:55.496752 9522 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:41171
I20260619 04:45:55.496775 9587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:41171 every 8 connection(s)
I20260619 04:45:55.498080 9522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:55.501780 9588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:55.507702 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 9522
I20260619 04:45:55.507903 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:55.508554 4046 external_mini_cluster.cc:1428] Setting key e247b87ff3ca2e001ac2f0fa714983fb
I20260619 04:45:55.512204 9588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0: Bootstrap starting.
I20260619 04:45:55.515563 9588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:55.516781 9588 log.cc:824] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:55.519621 9588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0: No bootstrap required, opened a new log
I20260619 04:45:55.522732 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:55.510440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49378 (local address 127.3.243.190:41171)
0619 04:45:55.511469 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:55.511480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:55.511522 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:45:55.513268 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:55.513289 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:55.513768 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:55.514109 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:55.514726 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:55.516191 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:55.517874 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:55.518553 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:55.518760 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:55.519909 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:55.519922 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:55.519942 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:45:55.520707 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:55.520808 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:45:55.520834 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:55.520957 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:55.521138 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 04:45:55.521459 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":380,"threads_started":1}
I20260619 04:45:55.526561 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:45:55.528808 9588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c34fddf2c4abaa68994665b4f22d0" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41171 } }
I20260619 04:45:55.529203 9588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:55.529335 9588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b6c34fddf2c4abaa68994665b4f22d0, State: Initialized, Role: FOLLOWER
I20260619 04:45:55.529989 9588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c34fddf2c4abaa68994665b4f22d0" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41171 } }
I20260619 04:45:55.530205 9588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:55.530357 9588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:55.530529 9588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:55.532704 9588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c34fddf2c4abaa68994665b4f22d0" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41171 } }
I20260619 04:45:55.533169 9588 leader_election.cc:302] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b6c34fddf2c4abaa68994665b4f22d0; no voters:
I20260619 04:45:55.533798 9588 leader_election.cc:288] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:55.534009 9593 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:55.534742 9593 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [term 1 LEADER]: Becoming Leader. State: Replica: 3b6c34fddf2c4abaa68994665b4f22d0, State: Running, Role: LEADER
I20260619 04:45:55.535315 9593 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c34fddf2c4abaa68994665b4f22d0" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41171 } }
I20260619 04:45:55.536069 9588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:55.538450 9595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b6c34fddf2c4abaa68994665b4f22d0. Latest consensus state: current_term: 1 leader_uuid: "3b6c34fddf2c4abaa68994665b4f22d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c34fddf2c4abaa68994665b4f22d0" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41171 } } }
I20260619 04:45:55.538445 9594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b6c34fddf2c4abaa68994665b4f22d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b6c34fddf2c4abaa68994665b4f22d0" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 41171 } } }
I20260619 04:45:55.538942 9594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:55.538834 9595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:55.540658 9602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:55.545482 9602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:55.560762 9602 catalog_manager.cc:1347] Generated new cluster ID: a469ad160f064528a3082b9b4f919ac6
I20260619 04:45:55.560981 9602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:55.590123 9602 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:55.591509 9602 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:55.598268 9602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b6c34fddf2c4abaa68994665b4f22d0: Generated new TSK 0
I20260619 04:45:55.599094 9602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:45:56.045926 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:55.615829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.3.243.190:41171)
0619 04:45:55.616034 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:55.616041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:55.616494 (+ 453us) server_negotiation.cc:407] Connection header received
0619 04:45:55.616684 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:55.616690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:55.616788 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:55.616948 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0619 04:45:55.617337 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:55.618459 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:55.619779 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:55.620504 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:55.620630 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:55.621089 (+ 459us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 04:45:56.045382 (+424293us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 04:45:56.045705 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49388: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 04:45:56.046099 9591 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49388: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 04:45:56.046315 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:45:55.615782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:41171 (local address 127.0.0.1:49388)
0619 04:45:55.616401 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:55.616416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:55.616532 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:55.616967 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:55.616970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:55.616978 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 04:45:55.617203 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:55.617210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:55.618585 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:55.618588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:55.619637 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:55.619647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:55.620660 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:55.620664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:55.620786 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:55.620818 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 04:45:56.045657 (+424839us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 04:45:56.045904 (+ 247us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.243.190:41171: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":364,"threads_started":1}
I20260619 04:45:56.047988 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 9522
I20260619 04:45:56.049031 9583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:45:56.151140 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 9522
2026-06-19T04:45:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1147 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (401 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (401 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:45:57 dist-test-slave-2qp8 krb5kdc[9632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:45:57 dist-test-slave-2qp8 krb5kdc[9632](info): set up 2 sockets
Jun 19 04:45:57 dist-test-slave-2qp8 krb5kdc[9632](info): commencing operation
krb5kdc: starting...
W20260619 04:45:58.044844 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:45:58 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:45:58.249293 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46505
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:46505
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:58.582316 9645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:58.582669 9645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:58.582839 9645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:58.590070 9645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:45:58.590186 9645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:58.590274 9645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:58.590375 9645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:45:58.590463 9645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:45:58.600554 9645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:46505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:58.601888 9645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:58.603184 9645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:58.615243 9650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:58.615293 9651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:58.615490 9653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:58.616266 9645 server_base.cc:1034] running on GCE node
I20260619 04:45:58.617061 9645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:58.618490 9645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:58.619714 9645 hybrid_clock.cc:648] HybridClock initialized: now 1781844358619668 us; error 48 us; skew 500 ppm
I20260619 04:45:58.620067 9645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:58 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844358, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:58.623410 9645 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:45:58.624989 9645 webserver.cc:460] Webserver started at http://127.3.243.190:35305/ using document root <none> and password file <none>
I20260619 04:45:58.625661 9645 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:58.625804 9645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:58.626157 9645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:58.629503 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "22cfb8f6185a48b9a6d306be1c89b001"
format_stamp: "Formatted at 2026-06-19 04:45:58 on dist-test-slave-2qp8"
server_key: "a17f1b6d8256f578394211a85b9861cb"
server_key_iv: "911630f14cf4a660f0f716d020d3df5d"
server_key_version: "encryptionkey@0"
I20260619 04:45:58.630246 9645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "22cfb8f6185a48b9a6d306be1c89b001"
format_stamp: "Formatted at 2026-06-19 04:45:58 on dist-test-slave-2qp8"
server_key: "a17f1b6d8256f578394211a85b9861cb"
server_key_iv: "911630f14cf4a660f0f716d020d3df5d"
server_key_version: "encryptionkey@0"
I20260619 04:45:58.639451 9645 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20260619 04:45:58.643661 9660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:58.645160 9645 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:45:58.645339 9645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "22cfb8f6185a48b9a6d306be1c89b001"
format_stamp: "Formatted at 2026-06-19 04:45:58 on dist-test-slave-2qp8"
server_key: "a17f1b6d8256f578394211a85b9861cb"
server_key_iv: "911630f14cf4a660f0f716d020d3df5d"
server_key_version: "encryptionkey@0"
I20260619 04:45:58.645541 9645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:58.666914 9645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:58.671264 9645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:58.671567 9645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:58.689656 9645 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:46505
I20260619 04:45:58.689708 9711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:46505 every 8 connection(s)
I20260619 04:45:58.691033 9645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:45:58.694703 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:45:58.698952 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 9645
I20260619 04:45:58.699175 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:45:58.699810 4046 external_mini_cluster.cc:1428] Setting key 8b553147a87cdf5213683b8271b24be1
I20260619 04:45:58.705535 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001: Bootstrap starting.
I20260619 04:45:58.709290 9712 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001: Neither blocks nor log segments found. Creating new log.
I20260619 04:45:58.710474 9712 log.cc:824] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001: Log is configured to *not* fsync() on all Append() calls
I20260619 04:45:58.713296 9712 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001: No bootstrap required, opened a new log
Jun 19 04:45:58 dist-test-slave-2qp8 krb5kdc[9632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:58.722966 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:58.701860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.3.243.190:46505)
0619 04:45:58.702731 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:58.702741 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:58.702782 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:45:58.704559 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:58.704582 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:58.704959 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:58.705282 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:58.707057 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:58.708426 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:58.710146 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:58.710887 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:58.711118 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:58.715549 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:58.715557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:58.715571 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:58.715608 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:58.718904 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:58.719696 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:58.719703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:58.719709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:58.719874 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:58.720266 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:58.720269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:58.720271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:58.720763 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:58.721003 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:58.721474 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 04:45:58.721858 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":417,"threads_started":1}
I20260619 04:45:58.722769 9712 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cfb8f6185a48b9a6d306be1c89b001" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46505 } }
I20260619 04:45:58.723192 9712 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:45:58.723344 9712 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22cfb8f6185a48b9a6d306be1c89b001, State: Initialized, Role: FOLLOWER
I20260619 04:45:58.723928 9712 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cfb8f6185a48b9a6d306be1c89b001" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46505 } }
I20260619 04:45:58.724246 9712 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:45:58.724397 9712 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:45:58.724602 9712 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:45:58.727079 9712 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cfb8f6185a48b9a6d306be1c89b001" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46505 } }
I20260619 04:45:58.727591 9712 leader_election.cc:302] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22cfb8f6185a48b9a6d306be1c89b001; no voters:
I20260619 04:45:58.728241 9712 leader_election.cc:288] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:45:58.728374 9717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:45:58.729607 9717 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [term 1 LEADER]: Becoming Leader. State: Replica: 22cfb8f6185a48b9a6d306be1c89b001, State: Running, Role: LEADER
I20260619 04:45:58.730094 9717 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cfb8f6185a48b9a6d306be1c89b001" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46505 } }
I20260619 04:45:58.730347 9712 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:45:58.734081 9719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22cfb8f6185a48b9a6d306be1c89b001. Latest consensus state: current_term: 1 leader_uuid: "22cfb8f6185a48b9a6d306be1c89b001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cfb8f6185a48b9a6d306be1c89b001" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46505 } } }
I20260619 04:45:58.734284 9718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22cfb8f6185a48b9a6d306be1c89b001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22cfb8f6185a48b9a6d306be1c89b001" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46505 } } }
I20260619 04:45:58.734552 9719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:58.734583 9718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001 [sys.catalog]: This master's current role is: LEADER
I20260619 04:45:58.736433 9726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:45:58.741426 9726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:45:58.749603 9726 catalog_manager.cc:1347] Generated new cluster ID: f0777fd8009d410e9eeda401572a1a28
I20260619 04:45:58.749706 9726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:45:58.761415 9726 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:45:58.762761 9726 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:45:58.774544 9726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22cfb8f6185a48b9a6d306be1c89b001: Generated new TSK 0
I20260619 04:45:58.775627 9726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:45:58.836380 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46505
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:59.159045 9738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:59.159355 9738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:59.159518 9738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:59.166610 9738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:59.166800 9738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:59.166971 9738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:45:59.176621 9738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:59.177901 9738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:59.179229 9738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:59.192543 9743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:59.192612 9744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:59.192591 9746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:59.193233 9738 server_base.cc:1034] running on GCE node
I20260619 04:45:59.193819 9738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:59.194865 9738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:59.196094 9738 hybrid_clock.cc:648] HybridClock initialized: now 1781844359196041 us; error 49 us; skew 500 ppm
I20260619 04:45:59.196460 9738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:59 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:59.202966 9738 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:45:59.204463 9738 webserver.cc:460] Webserver started at http://127.3.243.129:34275/ using document root <none> and password file <none>
I20260619 04:45:59.205148 9738 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:59.205287 9738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:59.205590 9738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:59.208525 9738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "f36256e8bea34ff78c7ff08de97de4df"
format_stamp: "Formatted at 2026-06-19 04:45:59 on dist-test-slave-2qp8"
server_key: "4100052ef08822717c02fe29d67c5d04"
server_key_iv: "43a9242c8a480b414a2d00072a342fb9"
server_key_version: "encryptionkey@0"
I20260619 04:45:59.209293 9738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "f36256e8bea34ff78c7ff08de97de4df"
format_stamp: "Formatted at 2026-06-19 04:45:59 on dist-test-slave-2qp8"
server_key: "4100052ef08822717c02fe29d67c5d04"
server_key_iv: "43a9242c8a480b414a2d00072a342fb9"
server_key_version: "encryptionkey@0"
I20260619 04:45:59.214399 9738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260619 04:45:59.217742 9753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:59.219007 9738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:45:59.219174 9738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "f36256e8bea34ff78c7ff08de97de4df"
format_stamp: "Formatted at 2026-06-19 04:45:59 on dist-test-slave-2qp8"
server_key: "4100052ef08822717c02fe29d67c5d04"
server_key_iv: "43a9242c8a480b414a2d00072a342fb9"
server_key_version: "encryptionkey@0"
I20260619 04:45:59.219365 9738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:59.238356 9738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:59.242288 9738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:59.242578 9738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:59.243489 9738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:59.244938 9738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:59.245049 9738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:59.245198 9738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:59.245275 9738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:59.277554 9738 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:38887
I20260619 04:45:59.277647 9865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:38887 every 8 connection(s)
I20260619 04:45:59.278867 9738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:45:59.288230 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 9738
I20260619 04:45:59.288427 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:45:59.289213 4046 external_mini_cluster.cc:1428] Setting key 6b2a2f04daa2085b5628d403fc56772e
Jun 19 04:45:59 dist-test-slave-2qp8 krb5kdc[9632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:59.299290 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:59.281470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:58191 (local address 127.3.243.190:46505)
0619 04:45:59.281910 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:59.281914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:59.282715 (+ 801us) server_negotiation.cc:407] Connection header received
0619 04:45:59.284565 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:59.284570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:59.284660 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:59.284777 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:59.286211 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:59.286897 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:59.288597 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:59.289277 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:59.289422 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:59.293348 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:59.293353 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:59.293356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:59.293386 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:59.295872 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:59.296825 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:59.296829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:59.296832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:59.296942 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:59.297410 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:59.297413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:59.297415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:59.297656 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:59.297765 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:59.298653 (+ 888us) server_negotiation.cc:299] Negotiation successful
0619 04:45:59.298819 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":245,"threads_started":1}
I20260619 04:45:59.300369 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:59.281866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:46505 (local address 127.3.243.129:58191)
0619 04:45:59.282558 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:59.282600 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:59.284298 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:59.285011 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:59.285019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:59.285512 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:59.286048 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:59.286064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:59.287055 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:59.287067 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:59.288404 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:59.288416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:59.289489 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:59.289494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:59.289674 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:59.290315 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:59.290357 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:59.293136 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:59.296037 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:59.296049 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:59.296067 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:59.296707 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:59.297059 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:59.297067 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:59.297073 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:59.297304 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:59.297788 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:59.297796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:59.298242 (+ 446us) client_negotiation.cc:769] Sending connection context
0619 04:45:59.298574 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 04:45:59.299142 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":274,"threads_started":1}
I20260619 04:45:59.301944 9866 heartbeater.cc:347] Connected to a master server at 127.3.243.190:46505
I20260619 04:45:59.302239 9866 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:59.302942 9866 heartbeater.cc:511] Master 127.3.243.190:46505 requested a full tablet report, sending...
I20260619 04:45:59.305014 9677 ts_manager.cc:194] Registered new tserver with Master: f36256e8bea34ff78c7ff08de97de4df (127.3.243.129:38887)
I20260619 04:45:59.306555 9677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:58191
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:45:59.335774 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46505
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:45:59.657333 9872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:45:59.657660 9872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:45:59.657825 9872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:45:59.665113 9872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:45:59.665262 9872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:45:59.665442 9872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:45:59.675088 9872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:45:59.676450 9872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:45:59.677906 9872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:45:59.690909 9878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:45:59.691861 9880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:59.692152 9872 server_base.cc:1034] running on GCE node
W20260619 04:45:59.692620 9877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:45:59.693353 9872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:45:59.694404 9872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:45:59.695629 9872 hybrid_clock.cc:648] HybridClock initialized: now 1781844359695573 us; error 53 us; skew 500 ppm
I20260619 04:45:59.695945 9872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:45:59 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:45:59.699203 9872 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:45:59.700758 9872 webserver.cc:460] Webserver started at http://127.3.243.130:42025/ using document root <none> and password file <none>
I20260619 04:45:59.701459 9872 fs_manager.cc:359] Metadata directory not provided
I20260619 04:45:59.701608 9872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:45:59.701908 9872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:45:59.704895 9872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "448d93b1e8164bcb87b4b0c522225846"
format_stamp: "Formatted at 2026-06-19 04:45:59 on dist-test-slave-2qp8"
server_key: "32c07e9af550fe26b804e160f13ca731"
server_key_iv: "4750359606b62abd4c31c69b551e5bd3"
server_key_version: "encryptionkey@0"
I20260619 04:45:59.705672 9872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "448d93b1e8164bcb87b4b0c522225846"
format_stamp: "Formatted at 2026-06-19 04:45:59 on dist-test-slave-2qp8"
server_key: "32c07e9af550fe26b804e160f13ca731"
server_key_iv: "4750359606b62abd4c31c69b551e5bd3"
server_key_version: "encryptionkey@0"
I20260619 04:45:59.710705 9872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:45:59.714120 9887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:59.715375 9872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:45:59.715564 9872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "448d93b1e8164bcb87b4b0c522225846"
format_stamp: "Formatted at 2026-06-19 04:45:59 on dist-test-slave-2qp8"
server_key: "32c07e9af550fe26b804e160f13ca731"
server_key_iv: "4750359606b62abd4c31c69b551e5bd3"
server_key_version: "encryptionkey@0"
I20260619 04:45:59.715776 9872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:45:59.751749 9872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:45:59.755857 9872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:45:59.756176 9872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:45:59.757143 9872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:45:59.758500 9872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:45:59.758625 9872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:59.758781 9872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:45:59.758852 9872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:45:59.791090 9872 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:43151
I20260619 04:45:59.791146 9999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:43151 every 8 connection(s)
I20260619 04:45:59.792502 9872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:45:59.795885 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 9872
I20260619 04:45:59.796089 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:45:59.796617 4046 external_mini_cluster.cc:1428] Setting key 18ea54b0df7ad40c922ecb4adb168d1b
Jun 19 04:45:59 dist-test-slave-2qp8 krb5kdc[9632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:45:59.813261 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:59.794971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:42385 (local address 127.3.243.190:46505)
0619 04:45:59.795184 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 04:45:59.795190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:45:59.796476 (+ 1286us) server_negotiation.cc:407] Connection header received
0619 04:45:59.798687 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:45:59.798692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:45:59.798772 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:45:59.798878 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:45:59.800339 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:59.801479 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:59.802995 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:45:59.803709 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:45:59.803843 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:59.807242 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:45:59.807247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:45:59.807250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:45:59.807281 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:45:59.809871 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:59.810581 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:59.810588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:59.810596 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:59.810771 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:45:59.811266 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:45:59.811274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:45:59.811284 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:45:59.811686 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:45:59.811851 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 04:45:59.812575 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 04:45:59.812849 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:45:59.813989 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:45:59.795499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:46505 (local address 127.3.243.130:42385)
0619 04:45:59.796297 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 04:45:59.796349 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:45:59.798358 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:45:59.799064 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:45:59.799072 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:45:59.799574 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:45:59.800177 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:59.800196 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:59.801642 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:59.801646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:59.802853 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:45:59.802863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:45:59.803932 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:45:59.803937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:45:59.804189 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:45:59.804867 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:45:59.804907 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:45:59.807028 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:45:59.810007 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:59.810017 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:59.810033 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:59.810450 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:59.810888 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:45:59.810892 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:45:59.810896 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:45:59.811076 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:45:59.811817 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:45:59.811822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:45:59.812171 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 04:45:59.812428 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 04:45:59.812902 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":334,"threads_started":1}
I20260619 04:45:59.815475 10000 heartbeater.cc:347] Connected to a master server at 127.3.243.190:46505
I20260619 04:45:59.815775 10000 heartbeater.cc:464] Registering TS with master...
I20260619 04:45:59.816588 10000 heartbeater.cc:511] Master 127.3.243.190:46505 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:45:59.818094 9677 ts_manager.cc:194] Registered new tserver with Master: 448d93b1e8164bcb87b4b0c522225846 (127.3.243.130:43151)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
I20260619 04:45:59.819126 9677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:42385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:45:59.843312 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46505
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:00.166837 10005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:00.167172 10005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:00.167348 10005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:00.174528 10005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:00.174672 10005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:00.174854 10005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:46:00.184664 10005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:46505
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:00.185928 10005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:00.187486 10005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:00.201886 10011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:00.201907 10010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:00.202193 10013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:00.202541 10005 server_base.cc:1034] running on GCE node
I20260619 04:46:00.203131 10005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:00.204017 10005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:00.205233 10005 hybrid_clock.cc:648] HybridClock initialized: now 1781844360205165 us; error 61 us; skew 500 ppm
I20260619 04:46:00.205564 10005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:00.208748 10005 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:46:00.210067 10005 webserver.cc:460] Webserver started at http://127.3.243.131:41173/ using document root <none> and password file <none>
I20260619 04:46:00.210731 10005 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:00.210826 10005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:00.211145 10005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:00.214076 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "fb18c383836944b784d8aaa8a08f6615"
format_stamp: "Formatted at 2026-06-19 04:46:00 on dist-test-slave-2qp8"
server_key: "b28b09bbc1c533f19180e81af1651ce9"
server_key_iv: "d16dcaa1c5d9d3a916b1d65b16cdd774"
server_key_version: "encryptionkey@0"
I20260619 04:46:00.214830 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "fb18c383836944b784d8aaa8a08f6615"
format_stamp: "Formatted at 2026-06-19 04:46:00 on dist-test-slave-2qp8"
server_key: "b28b09bbc1c533f19180e81af1651ce9"
server_key_iv: "d16dcaa1c5d9d3a916b1d65b16cdd774"
server_key_version: "encryptionkey@0"
I20260619 04:46:00.219792 10005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:46:00.223253 10020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:00.224658 10005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:46:00.224828 10005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "fb18c383836944b784d8aaa8a08f6615"
format_stamp: "Formatted at 2026-06-19 04:46:00 on dist-test-slave-2qp8"
server_key: "b28b09bbc1c533f19180e81af1651ce9"
server_key_iv: "d16dcaa1c5d9d3a916b1d65b16cdd774"
server_key_version: "encryptionkey@0"
I20260619 04:46:00.225020 10005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:00.241926 10005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:00.245982 10005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:00.246272 10005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:00.247243 10005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:00.248579 10005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:00.248677 10005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:00.248837 10005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:00.248898 10005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:00.285100 10005 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:44993
I20260619 04:46:00.285142 10132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:44993 every 8 connection(s)
I20260619 04:46:00.286505 10005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:46:00.292887 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 10005
I20260619 04:46:00.293072 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:46:00.293524 4046 external_mini_cluster.cc:1428] Setting key 98a12391ebef19dbbbaac230db4f36c3
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:46:00.305907 9868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:00.289146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:48221 (local address 127.3.243.190:46505)
0619 04:46:00.289330 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.289335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.290430 (+ 1095us) server_negotiation.cc:407] Connection header received
0619 04:46:00.292247 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.292253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.292356 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.292468 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.294178 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.295261 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.297117 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.297554 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.297654 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:00.300929 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:00.300934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:00.300937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:00.300967 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:00.303358 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.303978 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.303981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.303984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.304186 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.304551 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.304554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.304557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.304799 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:00.304906 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:00.305522 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 04:46:00.305676 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:46:00.306982 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:00.289570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:46505 (local address 127.3.243.131:48221)
0619 04:46:00.290273 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:00.290310 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:00.291938 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:00.292719 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:00.292727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:00.293345 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:00.293941 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:00.293958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.295443 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.295451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:00.296981 (+ 1530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:00.296999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.297735 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.297740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:00.297922 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:00.298633 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:00.298660 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:00.300756 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:00.303474 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:00.303480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:00.303491 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:00.303875 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:00.304281 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:00.304284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:00.304286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:00.304453 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:00.304897 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:00.304902 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:00.305186 (+ 284us) client_negotiation.cc:769] Sending connection context
0619 04:46:00.305414 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 04:46:00.305769 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":278,"threads_started":1}
I20260619 04:46:00.308578 10133 heartbeater.cc:347] Connected to a master server at 127.3.243.190:46505
I20260619 04:46:00.308864 10133 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:00.309396 9866 heartbeater.cc:503] Master 127.3.243.190:46505 was elected leader, sending a full tablet report...
I20260619 04:46:00.309763 10133 heartbeater.cc:511] Master 127.3.243.190:46505 requested a full tablet report, sending...
I20260619 04:46:00.311064 9677 ts_manager.cc:194] Registered new tserver with Master: fb18c383836944b784d8aaa8a08f6615 (127.3.243.131:44993)
I20260619 04:46:00.311801 9677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:48221
I20260619 04:46:00.321280 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/test-admin.keytab.
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:00.348747 4046 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 04:46:00.354128 10140 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:00.351581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.3.243.129:38887)
0619 04:46:00.352284 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.352292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.352316 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:46:00.352374 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.352381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.352597 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.352773 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.353643 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37438: BlockingRecv error: recv got EOF from 127.0.0.1:37438 (error 108)
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":300,"threads_started":1}
W20260619 04:46:00.354421 4046 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc)
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:00.355948 4046 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:00.357285 4046 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:46:00.371871 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:00.358851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.3.243.129:38887)
0619 04:46:00.358987 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.358991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.359064 (+ 73us) server_negotiation.cc:407] Connection header received
0619 04:46:00.359236 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.359240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.359306 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.359436 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.360390 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.361356 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.362516 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.363060 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.363151 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:00.366712 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:00.366723 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:00.366737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:00.366768 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:00.369440 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.370023 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.370029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.370032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.370143 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.370494 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.370497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.370499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.370827 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:00.371044 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:00.371303 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 04:46:00.371579 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 04:46:00.377555 10140 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:00.375830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.3.243.129:38887)
0619 04:46:00.376057 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.376063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.376356 (+ 293us) server_negotiation.cc:407] Connection header received
0619 04:46:00.376480 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.376484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.376539 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.376613 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.377304 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37464: BlockingRecv error: recv got EOF from 127.0.0.1:37464 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260619 04:46:00.377691 4046 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc)
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:00.379136 4046 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:00.380416 4046 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:46:00.394454 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:00.381661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37468 (local address 127.3.243.129:38887)
0619 04:46:00.381861 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.381867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.382108 (+ 241us) server_negotiation.cc:407] Connection header received
0619 04:46:00.382274 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.382279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.382343 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.382438 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.383344 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.384029 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.385272 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.385713 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.385849 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:00.389365 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:00.389370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:00.389373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:00.389412 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:00.392015 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.392668 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.392675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.392683 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.392803 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.393177 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.393183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.393186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.393440 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:00.393721 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:00.393989 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 04:46:00.394182 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 04:46:00.398546 10140 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:00.397220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.3.243.129:38887)
0619 04:46:00.397400 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.397404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.397483 (+ 79us) server_negotiation.cc:407] Connection header received
0619 04:46:00.397593 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.397596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.397660 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.397734 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.398347 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37476: BlockingRecv error: recv got EOF from 127.0.0.1:37476 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260619 04:46:00.398702 4046 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781844312210132-4046-0/krb5kdc/krb5cc)
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:00.400246 4046 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:00.401505 4046 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:46:00.415151 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:00.402632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.3.243.129:38887)
0619 04:46:00.402811 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.402819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.403102 (+ 283us) server_negotiation.cc:407] Connection header received
0619 04:46:00.403267 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.403278 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.403328 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.403422 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.404412 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.405109 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.406285 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.406683 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.406808 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:00.410281 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:00.410285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:00.410288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:00.410318 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:00.412649 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.413259 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.413266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.413274 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.413417 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.413773 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.413779 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.413782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.414065 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:00.414389 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:00.414658 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 04:46:00.414856 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:46:00.416639 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 9738
I20260619 04:46:00.428215 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:00.417300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37500 (local address 127.3.243.129:38887)
0619 04:46:00.417435 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.417439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.417822 (+ 383us) server_negotiation.cc:407] Connection header received
0619 04:46:00.417938 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.417942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.417994 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.418070 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.418897 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.419576 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.420910 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.421313 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.421403 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:00.423278 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:00.423282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:00.423285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:00.423311 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:00.425839 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.426422 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.426426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.426428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.426578 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.426971 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.426974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.426975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.427250 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:00.427500 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:00.427745 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 04:46:00.427888 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 04:46:00.428772 9861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:00.511473 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 9738
I20260619 04:46:00.527818 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 9872
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:46:00.542686 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:00.528842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39366 (local address 127.3.243.130:43151)
0619 04:46:00.529396 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.529403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.529428 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:46:00.529488 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.529503 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.529701 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.529828 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.530792 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.531873 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.533094 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.533643 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.533737 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:00.537329 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:00.537339 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:00.537367 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:00.537405 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:00.540393 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.540971 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.540977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.540979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.541094 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.541400 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.541403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.541405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.541742 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:00.541949 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:00.542188 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 04:46:00.542373 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":312,"threads_started":1}
I20260619 04:46:00.543931 9995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:00.627046 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 9872
I20260619 04:46:00.642942 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 10005
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[9632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:46:00.659047 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:00.644063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.3.243.131:44993)
0619 04:46:00.644735 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:00.644742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:00.644771 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:46:00.644833 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:00.644839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:00.645065 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:00.645199 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:00.646120 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.647152 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.648445 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:00.649010 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:00.649106 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:00.653204 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:00.653215 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:00.653230 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:00.653264 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:00.655916 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.656887 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.656895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.656898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.657026 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:00.657461 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:00.657470 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:00.657472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:00.657853 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:00.658160 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:00.658454 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 04:46:00.658753 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":346,"threads_started":1}
I20260619 04:46:00.660387 10128 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:00.741379 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 10005
I20260619 04:46:00.756821 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 9645
I20260619 04:46:00.757933 9707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:00.862042 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 9645
2026-06-19T04:46:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3914 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[10160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[10160](info): set up 2 sockets
Jun 19 04:46:00 dist-test-slave-2qp8 krb5kdc[10160](info): commencing operation
krb5kdc: starting...
W20260619 04:46:01.944972 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:46:02 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:46:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.190.keytab.
Entry for principal oryx/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.190.keytab.
I20260619 04:46:02.153438 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:34723
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.190.keytab
--principal=oryx/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:34723
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:02.475844 10173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:02.476210 10173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:02.476369 10173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:02.483887 10173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:02.484035 10173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:02.484177 10173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:02.484284 10173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:02.484368 10173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:02.494403 10173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:34723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.190.keytab
--principal=oryx/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:34723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:02.495796 10173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:02.497223 10173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:02.508874 10181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:02.508852 10179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:02.508932 10178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:02.509713 10173 server_base.cc:1034] running on GCE node
I20260619 04:46:02.510486 10173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:02.511797 10173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:02.513037 10173 hybrid_clock.cc:648] HybridClock initialized: now 1781844362512994 us; error 49 us; skew 500 ppm
I20260619 04:46:02.513418 10173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:02 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:02.517019 10173 init.cc:377] Logged in from keytab as oryx/127.3.243.190@KRBTEST.COM (short username oryx)
I20260619 04:46:02.518684 10173 webserver.cc:460] Webserver started at http://127.3.243.190:41271/ using document root <none> and password file <none>
I20260619 04:46:02.519388 10173 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:02.519517 10173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:02.519843 10173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:02.522886 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "7ec97feb168a47ad82c99a5bab5e6ed8"
format_stamp: "Formatted at 2026-06-19 04:46:02 on dist-test-slave-2qp8"
server_key: "b90f1c59e6445c59e6253b8f3924d1b3"
server_key_iv: "61ce823c37bd7ed0db3adabd9739466e"
server_key_version: "encryptionkey@0"
I20260619 04:46:02.523641 10173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "7ec97feb168a47ad82c99a5bab5e6ed8"
format_stamp: "Formatted at 2026-06-19 04:46:02 on dist-test-slave-2qp8"
server_key: "b90f1c59e6445c59e6253b8f3924d1b3"
server_key_iv: "61ce823c37bd7ed0db3adabd9739466e"
server_key_version: "encryptionkey@0"
I20260619 04:46:02.528807 10173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:46:02.532228 10188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:02.533645 10173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:46:02.533773 10173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "7ec97feb168a47ad82c99a5bab5e6ed8"
format_stamp: "Formatted at 2026-06-19 04:46:02 on dist-test-slave-2qp8"
server_key: "b90f1c59e6445c59e6253b8f3924d1b3"
server_key_iv: "61ce823c37bd7ed0db3adabd9739466e"
server_key_version: "encryptionkey@0"
I20260619 04:46:02.533928 10173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:02.550729 10173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:02.555244 10173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:02.555477 10173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:02.573462 10173 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:34723
I20260619 04:46:02.573493 10239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:34723 every 8 connection(s)
I20260619 04:46:02.574833 10173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:46:02.578572 10240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:02.583573 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 10173
I20260619 04:46:02.583777 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:46:02.584550 4046 external_mini_cluster.cc:1428] Setting key 93253673cc6e7673cc0f11a5130efb99
I20260619 04:46:02.589377 10240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: Bootstrap starting.
I20260619 04:46:02.593111 10240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:02.594404 10240 log.cc:824] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:02.597280 10240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: No bootstrap required, opened a new log
Jun 19 04:46:02 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:02.604823 10240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } }
I20260619 04:46:02.605225 10240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:02.605363 10240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec97feb168a47ad82c99a5bab5e6ed8, State: Initialized, Role: FOLLOWER
I20260619 04:46:02.606030 10240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } }
I20260619 04:46:02.606267 10240 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:46:02.606418 10240 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:46:02.606594 10240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:02.607410 10243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:02.586500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.3.243.190:34723)
0619 04:46:02.587299 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:02.587312 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:02.587363 (+ 51us) server_negotiation.cc:407] Connection header received
0619 04:46:02.589045 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:02.589066 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:02.589451 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:02.589829 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:02.591649 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:02.593039 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:02.594830 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:02.595553 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:02.595767 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:02.599687 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:02.599697 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:02.599711 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:02.599753 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:02.602984 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:02.603688 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:02.603695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:02.603700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:02.603867 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:02.604323 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:02.604329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:02.604334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:02.604850 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:02.605155 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:02.605600 (+ 445us) server_negotiation.cc:299] Negotiation successful
0619 04:46:02.606150 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":371,"threads_started":1}
I20260619 04:46:02.609107 10240 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } }
I20260619 04:46:02.609593 10240 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec97feb168a47ad82c99a5bab5e6ed8; no voters:
I20260619 04:46:02.610244 10240 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:46:02.610349 10245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:02.610970 10245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 1 LEADER]: Becoming Leader. State: Replica: 7ec97feb168a47ad82c99a5bab5e6ed8, State: Running, Role: LEADER
I20260619 04:46:02.611573 10245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } }
I20260619 04:46:02.612433 10240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:46:02.615475 10247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec97feb168a47ad82c99a5bab5e6ed8. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } }
I20260619 04:46:02.615844 10247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:02.615828 10246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } }
I20260619 04:46:02.616247 10246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:02.620466 10255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:46:02.625578 10255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:46:02.634696 10255 catalog_manager.cc:1347] Generated new cluster ID: 65a5930847eb4d36b4f0872cb67076f5
I20260619 04:46:02.634819 10255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:46:02.660822 10255 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:46:02.662134 10255 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:46:02.678381 10255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: Generated new TSK 0
I20260619 04:46:02.679215 10255 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab.
Entry for principal oryx/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab.
I20260619 04:46:02.743773 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:34723
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab
--principal=oryx/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:03.090754 10266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:03.091054 10266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:03.091152 10266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:03.098032 10266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:03.098137 10266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:03.098243 10266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:46:03.108059 10266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab
--principal=oryx/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:34723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:03.109735 10266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:03.111019 10266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:03.124723 10272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:03.124737 10271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:03.125216 10274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:03.125537 10266 server_base.cc:1034] running on GCE node
I20260619 04:46:03.126101 10266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:03.127002 10266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:03.128211 10266 hybrid_clock.cc:648] HybridClock initialized: now 1781844363128166 us; error 39 us; skew 500 ppm
I20260619 04:46:03.128516 10266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:03 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844363, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:03.131588 10266 init.cc:377] Logged in from keytab as oryx/127.3.243.129@KRBTEST.COM (short username oryx)
I20260619 04:46:03.132899 10266 webserver.cc:460] Webserver started at http://127.3.243.129:40839/ using document root <none> and password file <none>
I20260619 04:46:03.133523 10266 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:03.133616 10266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:03.133841 10266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:03.136742 10266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "464f118ea75742279b82ad32617a1967"
format_stamp: "Formatted at 2026-06-19 04:46:03 on dist-test-slave-2qp8"
server_key: "b2d6551ef138ff3a6004a9e777114ef4"
server_key_iv: "49a8234d25391982c341b0b47a0332fa"
server_key_version: "encryptionkey@0"
I20260619 04:46:03.137562 10266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "464f118ea75742279b82ad32617a1967"
format_stamp: "Formatted at 2026-06-19 04:46:03 on dist-test-slave-2qp8"
server_key: "b2d6551ef138ff3a6004a9e777114ef4"
server_key_iv: "49a8234d25391982c341b0b47a0332fa"
server_key_version: "encryptionkey@0"
I20260619 04:46:03.142660 10266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:46:03.146034 10281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:03.147289 10266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:46:03.147444 10266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "464f118ea75742279b82ad32617a1967"
format_stamp: "Formatted at 2026-06-19 04:46:03 on dist-test-slave-2qp8"
server_key: "b2d6551ef138ff3a6004a9e777114ef4"
server_key_iv: "49a8234d25391982c341b0b47a0332fa"
server_key_version: "encryptionkey@0"
I20260619 04:46:03.147605 10266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:03.159698 10266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:03.163880 10266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:03.164214 10266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:03.165145 10266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:03.166381 10266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:03.166453 10266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:03.166620 10266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:03.166692 10266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:03.200680 10266 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:34301
I20260619 04:46:03.200788 10393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:34301 every 8 connection(s)
I20260619 04:46:03.202039 10266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:46:03.206961 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 10266
I20260619 04:46:03.207217 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:46:03.208276 4046 external_mini_cluster.cc:1428] Setting key 98fc7f34db12d5104a2e83cd5d3b64de
Jun 19 04:46:03 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844363, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:03.224797 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:03.204906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:51351 (local address 127.3.243.190:34723)
0619 04:46:03.205405 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:03.205410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:03.206238 (+ 828us) server_negotiation.cc:407] Connection header received
0619 04:46:03.209294 (+ 3056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:03.209299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:03.209400 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:03.209536 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:03.211339 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:03.212043 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:03.213659 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:03.214133 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:03.214269 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:03.217691 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:03.217700 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:03.217708 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:03.217750 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:03.221558 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:03.222255 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:03.222263 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:03.222269 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:03.222416 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:03.222812 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:03.222819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:03.222825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:03.223195 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:03.223352 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:03.223983 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 04:46:03.224275 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":273,"threads_started":1}
I20260619 04:46:03.225404 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:03.205292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.129:51351)
0619 04:46:03.206042 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:03.206088 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:03.208992 (+ 2904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:03.209771 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:03.209779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:03.210363 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:03.211161 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:03.211188 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:03.212214 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:03.212222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:03.213505 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:03.213517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:03.214356 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:03.214363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:03.214592 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:03.215275 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:03.215296 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:03.217471 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:03.221700 (+ 4229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:03.221707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:03.221718 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:03.222135 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:03.222536 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:03.222539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:03.222542 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:03.222707 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:03.223312 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:03.223318 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:03.223622 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 04:46:03.223835 (+ 213us) client_negotiation.cc:240] Negotiation successful
0619 04:46:03.224290 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":291,"threads_started":1}
I20260619 04:46:03.227044 10394 heartbeater.cc:347] Connected to a master server at 127.3.243.190:34723
I20260619 04:46:03.227350 10394 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:03.228061 10394 heartbeater.cc:511] Master 127.3.243.190:34723 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:46:03.230273 10205 ts_manager.cc:194] Registered new tserver with Master: 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301)
I20260619 04:46:03.231640 10205 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:51351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab.
Entry for principal oryx/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab.
I20260619 04:46:03.259313 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:34723
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab
--principal=oryx/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:03.575845 10400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:03.576193 10400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:03.576333 10400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:03.587888 10400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:03.588047 10400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:03.588255 10400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:46:03.601019 10400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab
--principal=oryx/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:34723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:03.602260 10400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:03.603508 10400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:03.616595 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:03.616703 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:03.616616 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:03.618633 10400 server_base.cc:1034] running on GCE node
I20260619 04:46:03.619264 10400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:03.620329 10400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:03.621524 10400 hybrid_clock.cc:648] HybridClock initialized: now 1781844363621465 us; error 50 us; skew 500 ppm
I20260619 04:46:03.621848 10400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:03 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844363, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:03.625052 10400 init.cc:377] Logged in from keytab as oryx/127.3.243.130@KRBTEST.COM (short username oryx)
I20260619 04:46:03.626497 10400 webserver.cc:460] Webserver started at http://127.3.243.130:34081/ using document root <none> and password file <none>
I20260619 04:46:03.627205 10400 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:03.627348 10400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:03.627645 10400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:03.630658 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "91cb2bca3e284632bf8cd12eacfb2819"
format_stamp: "Formatted at 2026-06-19 04:46:03 on dist-test-slave-2qp8"
server_key: "79788dbc7043d1c992767e7fd1208dd8"
server_key_iv: "576bf7fa97a41cbd5cc44b2a449c3e3a"
server_key_version: "encryptionkey@0"
I20260619 04:46:03.631431 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "91cb2bca3e284632bf8cd12eacfb2819"
format_stamp: "Formatted at 2026-06-19 04:46:03 on dist-test-slave-2qp8"
server_key: "79788dbc7043d1c992767e7fd1208dd8"
server_key_iv: "576bf7fa97a41cbd5cc44b2a449c3e3a"
server_key_version: "encryptionkey@0"
I20260619 04:46:03.636638 10400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 04:46:03.639958 10415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:03.641268 10400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:46:03.641431 10400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "91cb2bca3e284632bf8cd12eacfb2819"
format_stamp: "Formatted at 2026-06-19 04:46:03 on dist-test-slave-2qp8"
server_key: "79788dbc7043d1c992767e7fd1208dd8"
server_key_iv: "576bf7fa97a41cbd5cc44b2a449c3e3a"
server_key_version: "encryptionkey@0"
I20260619 04:46:03.641623 10400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:03.657791 10400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:03.661856 10400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:03.662163 10400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:03.663134 10400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:03.664554 10400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:03.664654 10400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:03.664808 10400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:03.664882 10400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:03.698076 10400 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:40913
I20260619 04:46:03.698120 10527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:40913 every 8 connection(s)
I20260619 04:46:03.699398 10400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:46:03.699937 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 10400
I20260619 04:46:03.700156 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:46:03.700793 4046 external_mini_cluster.cc:1428] Setting key 5352a7965a69fbe3b85c5455fb0aa7f2
Jun 19 04:46:03 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844363, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:03.724488 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:03.702345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:44313 (local address 127.3.243.190:34723)
0619 04:46:03.702551 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:03.702557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:03.704079 (+ 1522us) server_negotiation.cc:407] Connection header received
0619 04:46:03.706187 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:03.706195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:03.706286 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:03.706402 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:03.708526 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:03.709550 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:03.711169 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:03.711621 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:03.711760 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:03.716370 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:03.716381 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:03.716389 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:03.716447 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:03.720784 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:03.721698 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:03.721709 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:03.721715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:03.721916 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:03.722392 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:03.722398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:03.722403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:03.722808 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:03.723007 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:03.723715 (+ 708us) server_negotiation.cc:299] Negotiation successful
0619 04:46:03.723995 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for oryx/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:46:03.725168 10530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:03.703063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.130:44313)
0619 04:46:03.703906 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:03.703949 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:03.705857 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:03.706643 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:03.706656 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:03.707387 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:03.708321 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:03.708349 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:03.709717 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:03.709722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:03.711005 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:03.711017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:03.711862 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:03.711870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:03.712217 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:03.713263 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:03.713287 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:03.715908 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:03.721028 (+ 5120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:03.721041 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:03.721059 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:03.721531 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:03.722051 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:03.722054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:03.722056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:03.722243 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:03.722957 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:03.722963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:03.723297 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 04:46:03.723536 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 04:46:03.723932 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":338,"threads_started":1}
I20260619 04:46:03.726786 10528 heartbeater.cc:347] Connected to a master server at 127.3.243.190:34723
I20260619 04:46:03.727098 10528 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.131@KRBTEST.COM" created.
I20260619 04:46:03.727962 10528 heartbeater.cc:511] Master 127.3.243.190:34723 requested a full tablet report, sending...
I20260619 04:46:03.729606 10205 ts_manager.cc:194] Registered new tserver with Master: 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130:40913)
I20260619 04:46:03.730440 10205 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:44313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab.
Entry for principal oryx/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab.
I20260619 04:46:03.755106 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:34723
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab
--principal=oryx/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:04.082813 10533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:04.083156 10533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:04.083317 10533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:04.090320 10533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:04.090468 10533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:04.090648 10533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:46:04.100891 10533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab
--principal=oryx/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:34723
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:04.102238 10533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:04.103580 10533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:04.116878 10541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:04.116900 10539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:04.117136 10538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:04.117877 10533 server_base.cc:1034] running on GCE node
I20260619 04:46:04.118510 10533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:04.119448 10533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:04.120674 10533 hybrid_clock.cc:648] HybridClock initialized: now 1781844364120603 us; error 60 us; skew 500 ppm
I20260619 04:46:04.121035 10533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:04 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844364, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:04.124032 10533 init.cc:377] Logged in from keytab as oryx/127.3.243.131@KRBTEST.COM (short username oryx)
I20260619 04:46:04.125435 10533 webserver.cc:460] Webserver started at http://127.3.243.131:42121/ using document root <none> and password file <none>
I20260619 04:46:04.126092 10533 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:04.126187 10533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:04.126420 10533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:04.129300 10533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "e19e388de3464493a9e58d8ca26e8733"
format_stamp: "Formatted at 2026-06-19 04:46:04 on dist-test-slave-2qp8"
server_key: "04fbaad78ca65dca19ce8ec19ed5caa1"
server_key_iv: "9acb9458ec78278482c8a885daadf838"
server_key_version: "encryptionkey@0"
I20260619 04:46:04.130021 10533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "e19e388de3464493a9e58d8ca26e8733"
format_stamp: "Formatted at 2026-06-19 04:46:04 on dist-test-slave-2qp8"
server_key: "04fbaad78ca65dca19ce8ec19ed5caa1"
server_key_iv: "9acb9458ec78278482c8a885daadf838"
server_key_version: "encryptionkey@0"
I20260619 04:46:04.135102 10533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:46:04.138639 10548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:04.139969 10533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:46:04.140175 10533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "e19e388de3464493a9e58d8ca26e8733"
format_stamp: "Formatted at 2026-06-19 04:46:04 on dist-test-slave-2qp8"
server_key: "04fbaad78ca65dca19ce8ec19ed5caa1"
server_key_iv: "9acb9458ec78278482c8a885daadf838"
server_key_version: "encryptionkey@0"
I20260619 04:46:04.140359 10533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:04.156282 10533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:04.160089 10533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:04.160362 10533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:04.161267 10533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:04.162564 10533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:04.162634 10533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:04.162731 10533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:04.162774 10533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:04.199939 10533 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:44163
I20260619 04:46:04.200027 10660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:44163 every 8 connection(s)
I20260619 04:46:04.201344 10533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:46:04.205778 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 10533
I20260619 04:46:04.205957 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:46:04.206396 4046 external_mini_cluster.cc:1428] Setting key 2ed180fda68c77e033e4a4ebb4ffe08b
Jun 19 04:46:04 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844364, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:04.227002 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.204027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:53117 (local address 127.3.243.190:34723)
0619 04:46:04.204257 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:04.204263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:04.205527 (+ 1264us) server_negotiation.cc:407] Connection header received
0619 04:46:04.207734 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:04.207740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:04.207840 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:04.207946 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:04.209438 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.210199 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.211920 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.212416 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.212570 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.221553 (+ 8983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:04.221559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:04.221564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:04.221604 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:04.223904 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.224663 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.224668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.224672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.224804 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.225231 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.225236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.225239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.225526 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:04.225681 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:04.226407 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 04:46:04.226648 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 04:46:04.228040 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.204535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.131:53117)
0619 04:46:04.205354 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:04.205398 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:04.207436 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:04.208240 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:04.208249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:04.208746 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:04.209282 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.209298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.210339 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.210345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.211779 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.211791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.212675 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.212682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.212911 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.218398 (+ 5487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:04.218428 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:04.221318 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:04.224058 (+ 2740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.224068 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.224086 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.224532 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.224913 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.224918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.224922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.225116 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.225651 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:04.225659 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:04.226035 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 04:46:04.226309 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 04:46:04.226857 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":341,"threads_started":1}
I20260619 04:46:04.229547 10661 heartbeater.cc:347] Connected to a master server at 127.3.243.190:34723
I20260619 04:46:04.229816 10661 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:04.230520 10661 heartbeater.cc:511] Master 127.3.243.190:34723 requested a full tablet report, sending...
I20260619 04:46:04.231894 10205 ts_manager.cc:194] Registered new tserver with Master: e19e388de3464493a9e58d8ca26e8733 (127.3.243.131:44163)
I20260619 04:46:04.233024 10205 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:53117
I20260619 04:46:04.234129 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:46:04.235339 10394 heartbeater.cc:503] Master 127.3.243.190:34723 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.3.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.189.keytab.
Entry for principal oryx/127.3.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.189.keytab.
I20260619 04:46:04.281746 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.189:45177
--webserver_interface=127.3.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.189.keytab
--principal=oryx/127.3.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:34723,127.3.243.189:45177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:04.682526 10666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:04.682834 10666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:04.683007 10666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:04.690315 10666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:04.690426 10666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:04.690472 10666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:04.690568 10666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:04.690650 10666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:04.702656 10666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:34723,127.3.243.189:45177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.189.keytab
--principal=oryx/127.3.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.189:45177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
W20260619 04:46:04.703063 10666 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.243.190:34723,127.3.243.189:45177'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 04:46:04.704388 10666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:04.705700 10666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:04.717782 10671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:04.717743 10674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:04.717826 10672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:04.721592 10666 server_base.cc:1034] running on GCE node
I20260619 04:46:04.722479 10666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:04.723845 10666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:04.725054 10666 hybrid_clock.cc:648] HybridClock initialized: now 1781844364724965 us; error 61 us; skew 500 ppm
I20260619 04:46:04.725340 10666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:04 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844364, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:04.728439 10666 init.cc:377] Logged in from keytab as oryx/127.3.243.189@KRBTEST.COM (short username oryx)
I20260619 04:46:04.729928 10666 webserver.cc:460] Webserver started at http://127.3.243.189:40621/ using document root <none> and password file <none>
I20260619 04:46:04.730589 10666 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:04.730674 10666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:04.730918 10666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:04.733812 10528 heartbeater.cc:503] Master 127.3.243.190:34723 was elected leader, sending a full tablet report...
I20260619 04:46:04.733843 10666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data/instance:
uuid: "b318ae68e78d4f35ab54cfdfa7248430"
format_stamp: "Formatted at 2026-06-19 04:46:04 on dist-test-slave-2qp8"
server_key: "fc868a2e5938bde7dff625af47344a43"
server_key_iv: "0529da7e0414dbc4592c5d5aa70e1a70"
server_key_version: "encryptionkey@0"
I20260619 04:46:04.734714 10666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal/instance:
uuid: "b318ae68e78d4f35ab54cfdfa7248430"
format_stamp: "Formatted at 2026-06-19 04:46:04 on dist-test-slave-2qp8"
server_key: "fc868a2e5938bde7dff625af47344a43"
server_key_iv: "0529da7e0414dbc4592c5d5aa70e1a70"
server_key_version: "encryptionkey@0"
I20260619 04:46:04.739820 10666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 04:46:04.743414 10681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:04.744987 10666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:46:04.745179 10666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal
uuid: "b318ae68e78d4f35ab54cfdfa7248430"
format_stamp: "Formatted at 2026-06-19 04:46:04 on dist-test-slave-2qp8"
server_key: "fc868a2e5938bde7dff625af47344a43"
server_key_iv: "0529da7e0414dbc4592c5d5aa70e1a70"
server_key_version: "encryptionkey@0"
I20260619 04:46:04.745414 10666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:04.777601 10666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:04.781992 10666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:04.782280 10666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:04.800365 10666 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.189:45177
I20260619 04:46:04.800392 10732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.189:45177 every 8 connection(s)
I20260619 04:46:04.801771 10666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
I20260619 04:46:04.805397 10733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:04.805712 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 10666
I20260619 04:46:04.814350 10733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:46:04 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.189@KRBTEST.COM
Jun 19 04:46:04 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844364, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.189@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:04.831197 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.808259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.3.243.189:45177)
0619 04:46:04.809089 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:04.809101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:04.809140 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:46:04.811255 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:04.811277 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:04.811786 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:04.812202 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:04.813958 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.815292 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.817415 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.818326 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.818450 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.823165 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:04.823176 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:04.823192 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:04.823236 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:04.826588 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.827302 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.827313 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.827324 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.827525 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.827947 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.827954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.827957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.828788 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:04.829082 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:04.829463 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 04:46:04.829899 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":399,"threads_started":1}
I20260619 04:46:04.832967 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.816999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.3.243.190:34723)
0619 04:46:04.817464 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:04.817469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:04.817753 (+ 284us) server_negotiation.cc:407] Connection header received
0619 04:46:04.817948 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:04.817953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:04.818036 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:04.818138 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:04.819664 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.820460 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.822214 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.822865 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.823032 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.826905 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:04.826910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:04.826913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:04.826944 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:04.830212 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.830861 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.830866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.830869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.831023 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.831408 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.831413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.831417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.831737 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:04.831891 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:04.832393 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 04:46:04.832613 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":248,"threads_started":1}
I20260619 04:46:04.833055 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.816957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:43406)
0619 04:46:04.817621 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:04.817660 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:04.817809 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:04.818205 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:04.818210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:04.819089 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:04.819498 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.819533 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.820619 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.820625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.822048 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.822064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.823123 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.823133 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.823328 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.823998 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:04.824032 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:04.826732 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:04.830333 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.830342 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.830346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.830749 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.831129 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.831132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.831134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.831262 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.831870 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:04.831877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:04.832051 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 04:46:04.832296 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 04:46:04.832605 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":302,"threads_started":1}
I20260619 04:46:04.834688 10733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:46:04 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844364, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.189@KRBTEST.COM for oryx/127.3.243.189@KRBTEST.COM
I20260619 04:46:04.846336 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.835236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.0.0.1:43414)
0619 04:46:04.835427 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:04.835444 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:04.835541 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:04.835773 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:04.835776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:04.836054 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:04.836237 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.836244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.837047 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.837051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.837909 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.837919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.838591 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.838597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.838776 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.839224 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:04.839239 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:04.841118 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:04.843818 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.843821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.843824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.844555 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.844903 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.844906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.844908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.845037 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.845562 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:04.845567 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:04.845755 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 04:46:04.845858 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 04:46:04.846066 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 04:46:04.846451 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.835430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.3.243.189:45177)
0619 04:46:04.835575 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:04.835579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:04.835592 (+ 13us) server_negotiation.cc:407] Connection header received
0619 04:46:04.835626 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:04.835629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:04.835677 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:04.835745 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:04.836335 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.836945 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.838050 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.838466 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.838588 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.841279 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:04.841284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:04.841287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:04.841313 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:04.843715 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.844681 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.844685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.844687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.844813 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.845127 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.845130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.845132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.845431 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:04.845585 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:04.845950 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 04:46:04.846171 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":65}
I20260619 04:46:04.851095 10733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: Bootstrap starting.
I20260619 04:46:04.854292 10733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:04.855357 10733 log.cc:824] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:04.857978 10733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: No bootstrap required, opened a new log
I20260619 04:46:04.864972 10733 raft_consensus.cc:348] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } }
I20260619 04:46:04.865370 10733 raft_consensus.cc:374] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:04.865476 10733 raft_consensus.cc:729] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b318ae68e78d4f35ab54cfdfa7248430, State: Initialized, Role: FOLLOWER
I20260619 04:46:04.866070 10733 consensus_queue.cc:260] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } }
I20260619 04:46:04.866900 10740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } } }
I20260619 04:46:04.867187 10740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:46:04.867969 10733 sys_catalog.cc:564] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 04:46:04.876703 10751 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:46:04.876859 10751 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:46:04.878574 10666 master_runner.cc:428] Detected that this master b318ae68e78d4f35ab54cfdfa7248430 is joining an existing cluster
I20260619 04:46:04.878674 10666 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.243.189:45177
I20260619 04:46:04.899860 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.883298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.0.0.1:43426)
0619 04:46:04.883901 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:04.883916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:04.884092 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:04.884582 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:04.884586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:04.885295 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:04.885627 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.885639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.887115 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.887122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.889063 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.889079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.890422 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.890430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.890636 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.891256 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:04.891281 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:04.892567 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:04.896820 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.896831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.896838 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.897404 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.897807 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.897814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.897820 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.897973 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.898629 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:04.898637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:04.898793 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 04:46:04.898913 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 04:46:04.899240 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":28,"thread_start_us":291,"threads_started":1}
I20260619 04:46:04.901070 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.882380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.3.243.190:34723)
0619 04:46:04.882572 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:04.882580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:04.883163 (+ 583us) server_negotiation.cc:407] Connection header received
0619 04:46:04.883347 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:04.883354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:04.883450 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:04.883553 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:04.885213 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.886302 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.888514 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.889141 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.889262 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.891871 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:04.891877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:04.891882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:04.891922 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:04.897066 (+ 5144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.898038 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.898046 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.898051 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.898220 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.898657 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.898662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.898666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.899025 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:04.899179 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:04.900450 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0619 04:46:04.900703 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:46:04.901859 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.882400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:43422)
0619 04:46:04.883053 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:04.883076 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:04.883210 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:04.883751 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:04.883755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:04.884571 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:04.885029 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.885043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.886461 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.886466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.888328 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.888343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.889512 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.889519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.889721 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.890408 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:04.890430 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:04.891641 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:04.897223 (+ 5582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.897228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.897232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.897869 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.898355 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.898360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.898364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.898527 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.899244 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:04.899249 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:04.899450 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 04:46:04.899621 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 04:46:04.901377 (+ 1756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":61,"thread_start_us":340,"threads_started":1}
I20260619 04:46:04.902068 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.883467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.3.243.189:45177)
0619 04:46:04.883690 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:04.883695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:04.883991 (+ 296us) server_negotiation.cc:407] Connection header received
0619 04:46:04.884263 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:04.884268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:04.884340 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:04.884402 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:04.885783 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.886894 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.889202 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.890013 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.890114 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.892748 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:04.892756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:04.892766 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:04.892799 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:04.896655 (+ 3856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.897533 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.897542 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.897548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.897684 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.898095 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.898101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.898107 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.898514 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:04.900152 (+ 1638us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:04.900284 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 04:46:04.901581 (+ 1297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1024,"server-negotiator.queue_time_us":68}
WARNING: no policy specified for oryx/127.3.243.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.188@KRBTEST.COM" created.
Jun 19 04:46:04 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.190@KRBTEST.COM for oryx/127.3.243.189@KRBTEST.COM
I20260619 04:46:04.927721 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.909439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.0.0.1:43430)
0619 04:46:04.910064 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:04.910112 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:04.910273 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:04.910790 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:04.910797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:04.911667 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:04.912329 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.912348 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.913548 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.913553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.915221 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:04.915236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.916384 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.916392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:04.916594 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.917415 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:04.917438 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:04.920400 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:04.924514 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.924527 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.924535 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.925084 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.925549 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:04.925556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:04.925560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:04.925736 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:04.926398 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:04.926406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:04.926650 (+ 244us) client_negotiation.cc:769] Sending connection context
0619 04:46:04.926836 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 04:46:04.927243 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":222,"threads_started":1}
I20260619 04:46:04.929260 10736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:04.909481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.3.243.189:45177)
0619 04:46:04.909691 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:04.909697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:04.910206 (+ 509us) server_negotiation.cc:407] Connection header received
0619 04:46:04.910496 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:04.910508 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:04.910621 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:04.910741 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:04.912525 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.913426 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.915391 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:04.916226 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:04.916364 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:04.920610 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:04.920619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:04.920624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:04.920662 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:04.924345 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.925256 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.925261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.925265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.925419 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:04.925878 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:04.925882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:04.925885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:04.926226 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:04.928327 (+ 2101us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:04.928464 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 04:46:04.928789 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:46:04.931432 10204 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.243.189:45177: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430"
member_type: NON_VOTER
last_known_addr {
host: "127.3.243.189"
port: 45177
}
attrs {
promote: true
}
}
dest_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8"
cas_config_opid_index: -1
I20260619 04:46:04.932718 10204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } }
I20260619 04:46:04.936203 10764 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b318ae68e78d4f35ab54cfdfa7248430 (127.3.243.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } }
I20260619 04:46:04.937424 10708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:04.940709 10764 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.243.189:45177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.188.keytab.
Entry for principal oryx/127.3.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.188.keytab.
I20260619 04:46:04.940871 10708 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Refusing update from remote peer 7ec97feb168a47ad82c99a5bab5e6ed8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 04:46:04.941120 10764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } }
I20260619 04:46:04.941323 10764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:04.942386 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.188:45313
--webserver_interface=127.3.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.188.keytab
--principal=oryx/127.3.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:34723,127.3.243.189:45177,127.3.243.188:45313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:46:04.944247 10666 master.cc:537] Master@127.3.243.189:45177 shutting down...
I20260619 04:46:04.944653 10763 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 04:46:04.950479 10192 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260619 04:46:04.955933 10192 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 -> Peer b318ae68e78d4f35ab54cfdfa7248430 (127.3.243.189:45177): Couldn't send request to peer b318ae68e78d4f35ab54cfdfa7248430. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 04:46:04.963135 10666 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:04.963599 10666 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:04.963824 10666 tablet_replica.cc:330] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: stopping tablet replica
I20260619 04:46:04.981671 10666 master.cc:552] Master@127.3.243.189:45177 shutdown complete.
I20260619 04:46:04.981781 10666 master_runner.cc:305] Clearing existing system tablet
I20260619 04:46:04.982200 10666 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:04.982295 10666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:04.985350 10666 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260619 04:46:04.989873 10769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:04.990268 10666 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:04.990535 10666 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260619 04:46:04.990662 10666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal
uuid: "b318ae68e78d4f35ab54cfdfa7248430"
format_stamp: "Formatted at 2026-06-19 04:46:04 on dist-test-slave-2qp8"
server_key: "fc868a2e5938bde7dff625af47344a43"
server_key_iv: "0529da7e0414dbc4592c5d5aa70e1a70"
server_key_version: "encryptionkey@0"
I20260619 04:46:04.995491 10666 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:05.002854 10666 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 04:46:05.003055 10666 log.cc:1196] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 04:46:05.003607 10666 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: Deleting consensus metadata
I20260619 04:46:05.003937 10666 master_runner.cc:315] Copying system tablet from 127.3.243.190:34723
I20260619 04:46:05.006680 10666 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: tablet copy: Beginning tablet copy session from remote peer at address 127.3.243.190:34723
I20260619 04:46:05.023952 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.008015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.3.243.190:34723)
0619 04:46:05.008233 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.008239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.008261 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:46:05.008522 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.008528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.008613 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.008728 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.010100 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.011262 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.013609 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.014238 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.014353 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.017064 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.017072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.017077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.017121 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.020784 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.021704 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.021710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.021714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.021874 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.022307 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.022312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.022316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.022696 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.022853 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.023321 (+ 468us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.023570 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:46:05.023977 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.007518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:43436)
0619 04:46:05.008097 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.008147 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.008308 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.008785 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.008793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.009491 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:05.009900 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.009918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.011443 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.011452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.013398 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.013415 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.014676 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.014682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.014862 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.015626 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:05.015648 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:05.016796 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:05.020950 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.020956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.020960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.021556 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.021996 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.022000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.022003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.022181 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.022827 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:05.022831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:05.023016 (+ 185us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.023174 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.023527 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":750,"threads_started":1}
I20260619 04:46:05.025390 10225 tablet_copy_service.cc:140] P 7ec97feb168a47ad82c99a5bab5e6ed8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b318ae68e78d4f35ab54cfdfa7248430 ({username='oryx', principal='oryx/127.3.243.189@KRBTEST.COM'} at 127.0.0.1:43436)
I20260619 04:46:05.025698 10225 tablet_copy_service.cc:161] P 7ec97feb168a47ad82c99a5bab5e6ed8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b318ae68e78d4f35ab54cfdfa7248430 at {username='oryx', principal='oryx/127.3.243.189@KRBTEST.COM'} at 127.0.0.1:43436: session id = b318ae68e78d4f35ab54cfdfa7248430-00000000000000000000000000000000
I20260619 04:46:05.030256 10225 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 04:46:05.032706 10666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 04:46:05.039973 10666 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: tablet copy: Starting download of 0 data blocks...
I20260619 04:46:05.040328 10666 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: tablet copy: Starting download of 1 WAL segments...
I20260619 04:46:05.043208 10666 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 04:46:05.048027 10225 tablet_copy_service.cc:342] P 7ec97feb168a47ad82c99a5bab5e6ed8: Request end of tablet copy session b318ae68e78d4f35ab54cfdfa7248430-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.243.189@KRBTEST.COM'} at 127.0.0.1:43436
I20260619 04:46:05.048278 10225 tablet_copy_service.cc:433] P 7ec97feb168a47ad82c99a5bab5e6ed8: ending tablet copy session b318ae68e78d4f35ab54cfdfa7248430-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b318ae68e78d4f35ab54cfdfa7248430
I20260619 04:46:05.067322 10666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:05.069875 10776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:05.076181 10777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:05.081024 10779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:05.082747 10666 server_base.cc:1034] running on GCE node
I20260619 04:46:05.083099 10666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:05.083734 10666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:05.085000 10666 hybrid_clock.cc:648] HybridClock initialized: now 1781844365084942 us; error 37 us; skew 500 ppm
I20260619 04:46:05.085215 10666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:05 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844365, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:05.087131 10666 init.cc:377] Logged in from keytab as oryx/127.3.243.189@KRBTEST.COM (short username oryx)
I20260619 04:46:05.088346 10666 webserver.cc:460] Webserver started at http://127.3.243.189:42735/ using document root <none> and password file <none>
I20260619 04:46:05.088671 10666 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:05.088759 10666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:05.091089 10666 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 04:46:05.092718 10786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:05.093133 10666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 04:46:05.093278 10666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal
uuid: "b318ae68e78d4f35ab54cfdfa7248430"
format_stamp: "Formatted at 2026-06-19 04:46:04 on dist-test-slave-2qp8"
server_key: "fc868a2e5938bde7dff625af47344a43"
server_key_iv: "0529da7e0414dbc4592c5d5aa70e1a70"
server_key_version: "encryptionkey@0"
I20260619 04:46:05.093436 10666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:05.128238 10666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:05.131650 10666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:05.154457 10666 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.189:45177
I20260619 04:46:05.154479 10837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.189:45177 every 8 connection(s)
I20260619 04:46:05.155416 10666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
I20260619 04:46:05.162577 10838 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:46:05.166872 10838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: Bootstrap starting.
I20260619 04:46:05.179641 10838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:46:05.180181 10838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: Bootstrap complete.
I20260619 04:46:05.181064 10838 raft_consensus.cc:348] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } }
I20260619 04:46:05.181303 10838 raft_consensus.cc:729] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b318ae68e78d4f35ab54cfdfa7248430, State: Initialized, Role: LEARNER
I20260619 04:46:05.181595 10838 consensus_queue.cc:260] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } }
I20260619 04:46:05.182533 10841 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } }
I20260619 04:46:05.182832 10838 sys_catalog.cc:564] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:46:05.182902 10841 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: This master's current role is: LEARNER
I20260619 04:46:05.197106 10852 catalog_manager.cc:1259] Loaded cluster ID: 65a5930847eb4d36b4f0872cb67076f5
I20260619 04:46:05.197225 10852 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: loading cluster ID for follower catalog manager: success
I20260619 04:46:05.200013 10852 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: acquiring CA information for follower catalog manager: success
I20260619 04:46:05.202598 10852 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:46:05.235782 10661 heartbeater.cc:503] Master 127.3.243.190:34723 was elected leader, sending a full tablet report...
W20260619 04:46:05.355715 10767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:05.356004 10767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:05.356096 10767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:05.363332 10767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:05.363443 10767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:05.363497 10767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:05.363557 10767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:05.363600 10767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:05.373867 10767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:34723,127.3.243.189:45177,127.3.243.188:45313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.188.keytab
--principal=oryx/127.3.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.188:45313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:05.375399 10767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:05.376910 10767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:05.389598 10858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:05.389688 10857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:05.389739 10860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:05.390466 10767 server_base.cc:1034] running on GCE node
I20260619 04:46:05.391165 10767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:05.392660 10767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:05.393875 10767 hybrid_clock.cc:648] HybridClock initialized: now 1781844365393847 us; error 52 us; skew 500 ppm
I20260619 04:46:05.394261 10767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:05 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844365, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:05.397435 10767 init.cc:377] Logged in from keytab as oryx/127.3.243.188@KRBTEST.COM (short username oryx)
I20260619 04:46:05.398835 10767 webserver.cc:460] Webserver started at http://127.3.243.188:44649/ using document root <none> and password file <none>
I20260619 04:46:05.399528 10767 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:05.399668 10767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:05.399960 10767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:05.402917 10767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data/instance:
uuid: "f80b1281aa2145729edb643968509225"
format_stamp: "Formatted at 2026-06-19 04:46:05 on dist-test-slave-2qp8"
server_key: "39b827e89b946976eb39b8475db4bf21"
server_key_iv: "fe815361e8f08df1e4d328a13ed271bd"
server_key_version: "encryptionkey@0"
I20260619 04:46:05.403684 10767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal/instance:
uuid: "f80b1281aa2145729edb643968509225"
format_stamp: "Formatted at 2026-06-19 04:46:05 on dist-test-slave-2qp8"
server_key: "39b827e89b946976eb39b8475db4bf21"
server_key_iv: "fe815361e8f08df1e4d328a13ed271bd"
server_key_version: "encryptionkey@0"
I20260619 04:46:05.409103 10767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260619 04:46:05.412714 10867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:05.414183 10767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 04:46:05.414350 10767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal
uuid: "f80b1281aa2145729edb643968509225"
format_stamp: "Formatted at 2026-06-19 04:46:05 on dist-test-slave-2qp8"
server_key: "39b827e89b946976eb39b8475db4bf21"
server_key_iv: "fe815361e8f08df1e4d328a13ed271bd"
server_key_version: "encryptionkey@0"
I20260619 04:46:05.414543 10767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:05.432178 10767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:05.436157 10767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:05.436430 10767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:05.448195 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.439557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.0.0.1:43432)
0619 04:46:05.440054 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.440071 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.440256 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.440900 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.440905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.440935 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:05.441166 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.441175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.443009 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.443015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.445315 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.445333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.447283 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.447294 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.447512 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.447582 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.447687 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.447815 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":261,"threads_started":1}
I20260619 04:46:05.449409 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.439788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.3.243.189:45177)
0619 04:46:05.440563 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.440567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.440581 (+ 14us) server_negotiation.cc:407] Connection header received
0619 04:46:05.440639 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.440643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.440713 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.440829 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:05.441345 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.442790 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.445510 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.446794 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.446970 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.447189 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.448742 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.448980 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":531,"threads_started":1}
I20260619 04:46:05.450496 10813 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 04:46:05.451071 10841 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec97feb168a47ad82c99a5bab5e6ed8. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } }
I20260619 04:46:05.451381 10841 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: This master's current role is: LEARNER
I20260619 04:46:05.460579 10767 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.188:45313
I20260619 04:46:05.460613 10920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.188:45313 every 8 connection(s)
I20260619 04:46:05.461932 10767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
I20260619 04:46:05.462945 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 10767
I20260619 04:46:05.466684 10921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:05.473389 10921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:46:05 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.188@KRBTEST.COM
Jun 19 04:46:05 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844365, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.188@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:05.484925 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.465245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.3.243.188:45313)
0619 04:46:05.466010 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.466023 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.466063 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:46:05.467721 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.467743 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.468183 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.468598 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.470023 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.471296 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.472629 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.473282 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.473466 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.477956 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.477963 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.477973 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.478011 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.480832 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.481478 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.481484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.481489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.481633 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.482042 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.482045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.482047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.482554 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.482858 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.483276 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.483690 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":326,"threads_started":1}
I20260619 04:46:05.489645 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.475746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:43440)
0619 04:46:05.476506 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.476551 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.476709 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.477099 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.477104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.477691 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:05.477971 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.477980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.479125 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.479130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.480405 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.480416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.481079 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.481085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.481241 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.481770 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:05.481785 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:05.484033 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:05.487339 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.487347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.487354 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.487880 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.488295 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.488298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.488300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.488413 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.488910 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:05.488915 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:05.489048 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.489185 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.489388 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":378,"threads_started":1}
I20260619 04:46:05.489713 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.475929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.3.243.190:34723)
0619 04:46:05.476166 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.476174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.476657 (+ 483us) server_negotiation.cc:407] Connection header received
0619 04:46:05.476840 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.476846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.476932 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.477032 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.478122 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.478982 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.480561 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.480917 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.481048 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.484255 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.484261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.484265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.484303 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.487218 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.487984 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.487988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.487990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.488200 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.488512 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.488515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.488517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.488801 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.488908 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.489278 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.489476 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 04:46:05.490823 10921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:46:05 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844365, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.188@KRBTEST.COM for oryx/127.3.243.189@KRBTEST.COM
I20260619 04:46:05.502753 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.491390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.0.0.1:43448)
0619 04:46:05.491556 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.491578 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.491646 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.492066 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.492070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.492414 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:05.492534 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.492541 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.493496 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.493503 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.494419 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.494429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.495143 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.495149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.495331 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.495792 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:05.495807 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:05.497777 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:05.500426 (+ 2649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.500431 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.500435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.500789 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.501152 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.501157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.501161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.501288 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.501796 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:05.501801 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:05.501951 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.502086 (+ 135us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.502411 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 04:46:05.502756 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.491618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.3.243.189:45177)
0619 04:46:05.491807 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.491812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.491827 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:46:05.491872 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.491876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.491932 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.492053 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.492664 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.493358 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.494551 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.495003 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.495117 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.497961 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.497966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.497969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.497999 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.500300 (+ 2301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.500915 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.500921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.500923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.501042 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.501396 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.501401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.501404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.501682 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.501830 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.502174 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.502413 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:46:05.503854 10921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:46:05 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844365, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.188@KRBTEST.COM for oryx/127.3.243.188@KRBTEST.COM
I20260619 04:46:05.515756 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.504753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:45313 (local address 127.0.0.1:51158)
0619 04:46:05.504925 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.504951 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.505069 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.505337 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.505341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.505690 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:05.505812 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.505820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.506736 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.506744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.507562 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.507572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.508300 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.508306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.508480 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.508974 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:05.508989 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:05.510816 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:05.513798 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.513802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.513804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.514171 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.514473 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.514477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.514479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.514614 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.515036 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:05.515039 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:05.515177 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.515285 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.515480 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 04:46:05.515954 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.504696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51158 (local address 127.3.243.188:45313)
0619 04:46:05.504888 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.504898 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.505014 (+ 116us) server_negotiation.cc:407] Connection header received
0619 04:46:05.505175 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.505184 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.505235 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.505323 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.505931 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.506607 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.507687 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.508066 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.508286 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.510974 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.510979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.510981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.511007 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.513681 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.514264 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.514268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.514270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.514377 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.514695 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.514698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.514700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.514934 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.515042 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.515355 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.515603 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":51}
I20260619 04:46:05.520584 10921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Bootstrap starting.
I20260619 04:46:05.523581 10921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:05.524652 10921 log.cc:824] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:05.527184 10921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: No bootstrap required, opened a new log
I20260619 04:46:05.534049 10921 raft_consensus.cc:348] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } }
I20260619 04:46:05.534395 10921 raft_consensus.cc:374] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:05.534515 10921 raft_consensus.cc:729] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f80b1281aa2145729edb643968509225, State: Initialized, Role: FOLLOWER
I20260619 04:46:05.535066 10921 consensus_queue.cc:260] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } }
I20260619 04:46:05.535872 10927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } } }
I20260619 04:46:05.536254 10927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:46:05.536964 10921 sys_catalog.cc:564] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 04:46:05.546264 10938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:46:05.546439 10938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:46:05.549692 10767 master_runner.cc:428] Detected that this master f80b1281aa2145729edb643968509225 is joining an existing cluster
I20260619 04:46:05.549829 10767 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.243.188:45313
I20260619 04:46:05.571689 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.553699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43450 (local address 127.3.243.190:34723)
0619 04:46:05.554050 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.554056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.554745 (+ 689us) server_negotiation.cc:407] Connection header received
0619 04:46:05.554923 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.554932 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.555033 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.555158 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.556550 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.557773 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.559966 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.560674 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.560779 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.563882 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.563890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.563896 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.563936 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.567520 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.569138 (+ 1618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.569144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.569147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.569303 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.569784 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.569789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.569793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.570144 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.570270 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.571013 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.571288 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227}
I20260619 04:46:05.572748 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.553792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:43450)
0619 04:46:05.554618 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.554638 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.554780 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.555192 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.555197 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.555904 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:05.556347 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.556363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.557994 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.557999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.559783 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.559798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.561113 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.561119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.561284 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.562023 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:05.562063 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:05.563644 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:05.567695 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.567712 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.567716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.568999 (+ 1283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.569456 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.569460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.569464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.569649 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.570369 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:05.570373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:05.570618 (+ 245us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.571902 (+ 1284us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.572252 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"mutex_wait_us":34,"thread_start_us":2534,"threads_started":1}
I20260619 04:46:05.578703 10875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.556799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.3.243.189:45177)
0619 04:46:05.556955 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.556961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.563668 (+ 6707us) server_negotiation.cc:407] Connection header received
0619 04:46:05.564081 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.564086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.564192 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.564260 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.565766 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.566878 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.568738 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.569632 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.569750 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.572223 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.572230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.572235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.572273 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.575435 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.576540 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.576545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.576548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.576682 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.577136 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.577141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.577144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.577470 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.577618 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.578140 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.578353 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:46:05.580070 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.556981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.0.0.1:43454)
0619 04:46:05.563568 (+ 6587us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.563585 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.563897 (+ 312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.564477 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.564487 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.565191 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:05.565587 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.565601 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.567066 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.567072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.568598 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.568608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.569987 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.569993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.570196 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.570806 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:05.570828 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:05.571986 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:05.575939 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.575943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.575946 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.576396 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.576807 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.576812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.576815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.577018 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.577695 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:05.577700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:05.577893 (+ 193us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.578974 (+ 1081us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.579388 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6447,"thread_start_us":6710,"threads_started":1}
I20260619 04:46:05.589412 10945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.553968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:45313 (local address 127.0.0.1:51164)
0619 04:46:05.556648 (+ 2680us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.556670 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.556802 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.557486 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.557494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.558436 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:05.558896 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.558911 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.560705 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.560712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.563127 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.563151 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.564509 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.564515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.564733 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.565393 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:05.565419 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:05.566555 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:05.570711 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.570718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.570725 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.571311 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.571740 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.571744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.572073 (+ 329us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.572264 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.588202 (+ 15938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:05.588209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:05.588407 (+ 198us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.588560 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.588924 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2545,"mutex_wait_us":3,"thread_start_us":349,"threads_started":1}
I20260619 04:46:05.589484 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.553842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.3.243.188:45313)
0619 04:46:05.554039 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.554046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.556731 (+ 2685us) server_negotiation.cc:407] Connection header received
0619 04:46:05.557063 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.557070 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.557157 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.557237 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.559070 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.560475 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.563317 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.564085 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.564200 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.566724 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.566730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.566734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.566764 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.570561 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.571429 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.571436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.571443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.571607 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.572396 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.572400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.572403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.572736 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.572894 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.588691 (+ 15797us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.589116 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":196,"server-negotiator.queue_time_us":36}
Jun 19 04:46:05 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.190@KRBTEST.COM for oryx/127.3.243.188@KRBTEST.COM
I20260619 04:46:05.591086 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 10266
I20260619 04:46:05.601649 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.578355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:45313 (local address 127.0.0.1:51174)
0619 04:46:05.578506 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.578525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.578631 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.581502 (+ 2871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.581507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.581942 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:05.582125 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.582136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.584910 (+ 2774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.584916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.586412 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.586427 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.587399 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.587405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.587623 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.588331 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:05.588355 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:05.591090 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:05.595657 (+ 4567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.595662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.595666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.596341 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.599535 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.599543 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.599548 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.599763 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.600574 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:05.600579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:05.600791 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.600964 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.601282 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260619 04:46:05.602862 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.578650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.3.243.188:45313)
0619 04:46:05.581147 (+ 2497us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.581152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.581174 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:46:05.581252 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.581257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.581331 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.582257 (+ 926us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.583478 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.584771 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.586565 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.587235 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.587347 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.591347 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.591355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.591359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.591393 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.595497 (+ 4104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.599156 (+ 3659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.599168 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.599172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.599359 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.599941 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.599948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.599954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.600400 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.602073 (+ 1673us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.602205 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.602433 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2349,"thread_start_us":2214,"threads_started":1}
I20260619 04:46:05.604372 10204 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.243.188:45313: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f80b1281aa2145729edb643968509225"
member_type: NON_VOTER
last_known_addr {
host: "127.3.243.188"
port: 45313
}
attrs {
promote: true
}
}
dest_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8"
cas_config_opid_index: 5
I20260619 04:46:05.605558 10204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } }
I20260619 04:46:05.607599 10763 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f80b1281aa2145729edb643968509225 (127.3.243.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.613372 10813 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 LEARNER]: Refusing update from remote peer 7ec97feb168a47ad82c99a5bab5e6ed8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 04:46:05.615159 10896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 04:46:05 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.129@KRBTEST.COM
I20260619 04:46:05.621304 10763 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.243.188:45313
I20260619 04:46:05.622118 10896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 FOLLOWER]: Refusing update from remote peer 7ec97feb168a47ad82c99a5bab5e6ed8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 04:46:05.623605 10767 master.cc:537] Master@127.3.243.188:45313 shutting down...
I20260619 04:46:05.626258 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.592685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.3.243.129:34301)
0619 04:46:05.603541 (+ 10856us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.603551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.603586 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:46:05.603660 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.603669 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.603988 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.604287 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.606043 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.607734 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.609829 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.610789 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.617005 (+ 6216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.617331 (+ 326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.617346 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.617365 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.617406 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.621705 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.622674 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.622684 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.622687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.622875 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.623332 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.623338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.623342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.623784 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.625260 (+ 1476us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.625468 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.625778 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10653,"thread_start_us":10619,"threads_started":1}
I20260619 04:46:05.628096 10389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:05.622014 10950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 04:46:05.644342 10950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260619 04:46:05.646059 10813 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f80b1281aa2145729edb643968509225 (127.3.243.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.650235 10841 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.650550 10841 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: This master's current role is: LEARNER
I20260619 04:46:05.652719 10763 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: attempting to promote NON_VOTER b318ae68e78d4f35ab54cfdfa7248430 to VOTER
I20260619 04:46:05.652688 10951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.653198 10951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:05.653816 10763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } }
I20260619 04:46:05.657431 10950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: NON_VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: true } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.657862 10950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:05.658335 10813 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 LEARNER]: Refusing update from remote peer 7ec97feb168a47ad82c99a5bab5e6ed8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 04:46:05.659077 10950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 04:46:05.660061 10767 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:05.660596 10767 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:05.660815 10767 tablet_replica.cc:330] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: stopping tablet replica
I20260619 04:46:05.662583 10812 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b318ae68e78d4f35ab54cfdfa7248430 (127.3.243.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.666877 10841 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.667152 10841 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:46:05.668931 10767 master.cc:552] Master@127.3.243.188:45313 shutdown complete.
I20260619 04:46:05.669979 10767 master_runner.cc:305] Clearing existing system tablet
I20260619 04:46:05.670583 10767 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:05.671196 10767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:05.673075 10949 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b318ae68e78d4f35ab54cfdfa7248430 (127.3.243.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.675866 10767 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:46:05.678258 10956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.678697 10956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:05.678989 10956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.679232 10956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:05.679971 10959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:05.681298 10767 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:05.681535 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.681061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:45313 (local address 127.0.0.1:51190)
0619 04:46:05.681223 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.681350 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:45313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 04:46:05.681699 10767 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.001s
I20260619 04:46:05.681841 10767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal
uuid: "f80b1281aa2145729edb643968509225"
format_stamp: "Formatted at 2026-06-19 04:46:05 on dist-test-slave-2qp8"
server_key: "39b827e89b946976eb39b8475db4bf21"
server_key_iv: "fe815361e8f08df1e4d328a13ed271bd"
server_key_version: "encryptionkey@0"
W20260619 04:46:05.682029 10191 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 -> Peer f80b1281aa2145729edb643968509225 (127.3.243.188:45313): Couldn't send request to peer f80b1281aa2145729edb643968509225. Status: Network error: Client connection negotiation failed: client connection to 127.3.243.188:45313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 04:46:05.687072 10767 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:05.693873 10767 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 04:46:05.694031 10767 log.cc:1196] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 04:46:05.694504 10767 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Deleting consensus metadata
I20260619 04:46:05.694751 10767 master_runner.cc:315] Copying system tablet from 127.3.243.190:34723
I20260619 04:46:05.697572 10767 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: tablet copy: Beginning tablet copy session from remote peer at address 127.3.243.190:34723
I20260619 04:46:05.718611 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.698812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.3.243.190:34723)
0619 04:46:05.699040 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:05.699046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:05.703976 (+ 4930us) server_negotiation.cc:407] Connection header received
0619 04:46:05.704197 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:05.704204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:05.704288 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:05.704405 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:05.705483 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.706633 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.708749 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.709415 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.709543 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.712036 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:05.712044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:05.712050 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:05.712091 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:05.715624 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.716438 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.716444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.716448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.716599 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:05.716996 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:05.717001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:05.717004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:05.717384 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:05.717529 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:05.718021 (+ 492us) server_negotiation.cc:299] Negotiation successful
0619 04:46:05.718266 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 04:46:05.718633 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:05.698346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:43460)
0619 04:46:05.703800 (+ 5454us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:05.703827 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:05.704011 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:05.704457 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:05.704463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:05.705026 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:05.705297 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.705313 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.706832 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.706837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.708599 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:05.708614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:05.709748 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:05.709755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:05.709932 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:05.710687 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:05.710709 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:05.711826 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:05.715761 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.715767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.715771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.716329 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.716710 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:05.716715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:05.716718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:05.716871 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:05.717504 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:05.717508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:05.717699 (+ 191us) client_negotiation.cc:769] Sending connection context
0619 04:46:05.717876 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 04:46:05.718192 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5280,"thread_start_us":5168,"threads_started":1}
I20260619 04:46:05.719561 10225 tablet_copy_service.cc:140] P 7ec97feb168a47ad82c99a5bab5e6ed8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f80b1281aa2145729edb643968509225 ({username='oryx', principal='oryx/127.3.243.188@KRBTEST.COM'} at 127.0.0.1:43460)
I20260619 04:46:05.719735 10225 tablet_copy_service.cc:161] P 7ec97feb168a47ad82c99a5bab5e6ed8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f80b1281aa2145729edb643968509225 at {username='oryx', principal='oryx/127.3.243.188@KRBTEST.COM'} at 127.0.0.1:43460: session id = f80b1281aa2145729edb643968509225-00000000000000000000000000000000
I20260619 04:46:05.723428 10225 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 04:46:05.725318 10767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 04:46:05.732264 10767 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: tablet copy: Starting download of 0 data blocks...
I20260619 04:46:05.732586 10767 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: tablet copy: Starting download of 1 WAL segments...
I20260619 04:46:05.735059 10767 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 04:46:05.739491 10225 tablet_copy_service.cc:342] P 7ec97feb168a47ad82c99a5bab5e6ed8: Request end of tablet copy session f80b1281aa2145729edb643968509225-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.243.188@KRBTEST.COM'} at 127.0.0.1:43460
I20260619 04:46:05.739625 10225 tablet_copy_service.cc:433] P 7ec97feb168a47ad82c99a5bab5e6ed8: ending tablet copy session f80b1281aa2145729edb643968509225-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f80b1281aa2145729edb643968509225
I20260619 04:46:05.752561 10767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:05.755452 10966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:05.757457 10967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:05.761771 10767 server_base.cc:1034] running on GCE node
W20260619 04:46:05.761863 10969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:05.762370 10767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:05.762981 10767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:05.764235 10767 hybrid_clock.cc:648] HybridClock initialized: now 1781844365764193 us; error 35 us; skew 500 ppm
I20260619 04:46:05.764539 10767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:05 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844365, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:05.766853 10767 init.cc:377] Logged in from keytab as oryx/127.3.243.188@KRBTEST.COM (short username oryx)
I20260619 04:46:05.768136 10767 webserver.cc:460] Webserver started at http://127.3.243.188:37933/ using document root <none> and password file <none>
I20260619 04:46:05.768453 10767 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:05.768548 10767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:05.771389 10767 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260619 04:46:05.773365 10976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:05.773864 10767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 04:46:05.774009 10767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal
uuid: "f80b1281aa2145729edb643968509225"
format_stamp: "Formatted at 2026-06-19 04:46:05 on dist-test-slave-2qp8"
server_key: "39b827e89b946976eb39b8475db4bf21"
server_key_iv: "fe815361e8f08df1e4d328a13ed271bd"
server_key_version: "encryptionkey@0"
I20260619 04:46:05.774202 10767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:05.786405 10767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:05.787045 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 10266
I20260619 04:46:05.791163 10767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:05.807711 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:34301
--local_ip_for_outbound_sockets=127.3.243.129
--tserver_master_addrs=127.3.243.190:34723
--webserver_port=40839
--webserver_interface=127.3.243.129
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab
--principal=oryx/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.243.190:34723,127.3.243.189:45177
--tserver_master_addrs=127.3.243.190:34723,127.3.243.189:45177,127.3.243.188:45313 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:46:05.826947 10767 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.188:45313
I20260619 04:46:05.826997 11028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.188:45313 every 8 connection(s)
I20260619 04:46:05.828081 10767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
I20260619 04:46:05.833860 11029 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:46:05.837986 11029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Bootstrap starting.
I20260619 04:46:05.856268 11029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:46:05.856799 11029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Bootstrap complete.
I20260619 04:46:05.857900 11029 raft_consensus.cc:348] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } }
I20260619 04:46:05.858192 11029 raft_consensus.cc:729] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f80b1281aa2145729edb643968509225, State: Initialized, Role: LEARNER
I20260619 04:46:05.858494 11029 consensus_queue.cc:260] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } }
I20260619 04:46:05.859723 11029 sys_catalog.cc:564] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:46:05.859402 11032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:05.860040 11032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: This master's current role is: LEARNER
I20260619 04:46:05.872941 11043 catalog_manager.cc:1259] Loaded cluster ID: 65a5930847eb4d36b4f0872cb67076f5
I20260619 04:46:05.873072 11043 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: loading cluster ID for follower catalog manager: success
I20260619 04:46:05.876652 11043 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: acquiring CA information for follower catalog manager: success
I20260619 04:46:05.879104 11043 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:46:06.128916 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.122813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.3.243.188:45313)
0619 04:46:06.123528 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:06.123533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:06.123549 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:46:06.123607 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:06.123610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:06.123683 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:06.123810 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:06.124311 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.125298 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.127022 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.127831 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.127994 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.128282 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:06.128452 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 04:46:06.128603 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":454,"threads_started":1}
I20260619 04:46:06.128894 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.122622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:45313 (local address 127.0.0.1:51194)
0619 04:46:06.122855 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:06.122879 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:06.123023 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:06.123831 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:06.123836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:06.123868 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:06.124094 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.124155 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.125442 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.125447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.126864 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.126876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.128007 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.128013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.128259 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.128297 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 04:46:06.128419 (+ 122us) client_negotiation.cc:240] Negotiation successful
0619 04:46:06.128576 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 04:46:06.129956 11003 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 04:46:06.130491 11032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec97feb168a47ad82c99a5bab5e6ed8. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:06.130796 11032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: This master's current role is: LEARNER
W20260619 04:46:06.179315 11010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:06.179734 11010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:06.179924 11010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:06.190311 11010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:06.190444 11010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:06.190621 11010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:46:06.200909 11010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab
--principal=oryx/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:34301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=40839
--tserver_master_addrs=127.3.243.190:34723,127.3.243.189:45177,127.3.243.188:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:06.202286 11010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:06.203760 11010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:06.217619 11050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:06.217716 11052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:06.217662 11049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:06.224536 11010 server_base.cc:1034] running on GCE node
I20260619 04:46:06.225170 11010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:06.226186 11010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:06.227411 11010 hybrid_clock.cc:648] HybridClock initialized: now 1781844366227368 us; error 44 us; skew 500 ppm
I20260619 04:46:06.227731 11010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844366, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:06.230890 11010 init.cc:377] Logged in from keytab as oryx/127.3.243.129@KRBTEST.COM (short username oryx)
I20260619 04:46:06.232331 11010 webserver.cc:460] Webserver started at http://127.3.243.129:40839/ using document root <none> and password file <none>
I20260619 04:46:06.233007 11010 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:06.233115 11010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:06.239001 11010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 04:46:06.241761 11059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:06.243141 11010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:46:06.243300 11010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "464f118ea75742279b82ad32617a1967"
format_stamp: "Formatted at 2026-06-19 04:46:03 on dist-test-slave-2qp8"
server_key: "b2d6551ef138ff3a6004a9e777114ef4"
server_key_iv: "49a8234d25391982c341b0b47a0332fa"
server_key_version: "encryptionkey@0"
I20260619 04:46:06.244504 11010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:06.259435 11010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:06.263469 11010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:06.263705 11010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:06.264658 11010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:06.265923 11010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:06.266006 11010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:06.266086 11010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:06.266142 11010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:06.301599 11010 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:34301
I20260619 04:46:06.301702 11171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:34301 every 8 connection(s)
I20260619 04:46:06.303078 11010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:46:06.312625 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 11010
I20260619 04:46:06.312978 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 10400
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844366, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844366, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.189@KRBTEST.COM
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844366, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.188@KRBTEST.COM
I20260619 04:46:06.332156 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.306322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:44391 (local address 127.3.243.188:45313)
0619 04:46:06.306508 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:06.306514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:06.308044 (+ 1530us) server_negotiation.cc:407] Connection header received
0619 04:46:06.310409 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:06.310415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:06.310506 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:06.310615 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:06.313662 (+ 3047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.314838 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.318989 (+ 4151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.319648 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.319745 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.325799 (+ 6054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:06.325807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:06.325814 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:06.325867 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:06.329212 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.329911 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.329917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.329922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.330055 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.330532 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.330539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.330543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.330893 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:06.331011 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:06.331543 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 04:46:06.331781 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:46:06.332793 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.308566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.3.243.129:41191)
0619 04:46:06.309157 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:06.309179 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:06.310161 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:06.311075 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:06.311087 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:06.312253 (+ 1166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:06.313441 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.313511 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.315838 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.315843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.317134 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.317146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.319191 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.319200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.319504 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.320293 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:06.320325 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:06.324891 (+ 4566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:06.328802 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.328810 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.328815 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.329346 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.329796 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.329802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.329807 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.329991 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.330739 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:06.330743 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:06.331292 (+ 549us) client_negotiation.cc:769] Sending connection context
0619 04:46:06.331454 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 04:46:06.331725 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":242,"spinlock_wait_cycles":15616,"thread_start_us":304,"threads_started":1}
I20260619 04:46:06.333021 11176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.306093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:35577 (local address 127.3.243.190:34723)
0619 04:46:06.306835 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:06.306844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:06.308036 (+ 1192us) server_negotiation.cc:407] Connection header received
0619 04:46:06.310502 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:06.310507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:06.310594 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:06.310698 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:06.314015 (+ 3317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.315087 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.317172 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.317817 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.317974 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.323103 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:06.323113 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:06.323121 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:06.323177 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:06.327141 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.328610 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.328615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.328619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.328775 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.329293 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.329297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.329301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.329715 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:06.329864 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:06.332435 (+ 2571us) server_negotiation.cc:299] Negotiation successful
0619 04:46:06.332669 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":517,"threads_started":1}
I20260619 04:46:06.333426 11177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.306881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.129:35577)
0619 04:46:06.307903 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:06.307923 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:06.310161 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:06.311075 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:06.311087 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:06.312253 (+ 1166us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:06.313441 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.313469 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.315212 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.315217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.317020 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.317031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.318049 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.318056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.318262 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.319072 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:06.319100 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:06.322796 (+ 3696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:06.327790 (+ 4994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.327799 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.327814 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.328463 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.328911 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.328917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.328921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.329160 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.329942 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:06.329949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:06.330399 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 04:46:06.330727 (+ 328us) client_negotiation.cc:240] Negotiation successful
0619 04:46:06.331272 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":221,"spinlock_wait_cycles":32000,"thread_start_us":416,"threads_started":1}
I20260619 04:46:06.333904 11180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.308730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:41191 (local address 127.3.243.189:45177)
0619 04:46:06.309320 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:06.309326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:06.309360 (+ 34us) server_negotiation.cc:407] Connection header received
0619 04:46:06.310502 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:06.310507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:06.310594 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:06.310698 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:06.313658 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.315660 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.317285 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.318192 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.318347 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.325196 (+ 6849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:06.325207 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:06.325215 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:06.325267 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:06.328659 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.329510 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.329516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.329520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.329663 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.330125 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.330131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.330134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.330515 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:06.330674 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:06.333281 (+ 2607us) server_negotiation.cc:299] Negotiation successful
0619 04:46:06.333511 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":362,"threads_started":1}
I20260619 04:46:06.334115 11178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.306882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:45313 (local address 127.3.243.129:44391)
0619 04:46:06.307869 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:06.307919 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:06.310123 (+ 2204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:06.311446 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:06.311449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:06.312218 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:06.313443 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.313469 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.315407 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.315415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.318774 (+ 3359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.318785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.320077 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.320085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.320313 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.321957 (+ 1644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:06.321979 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:06.325550 (+ 3571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:06.329350 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.329354 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.329357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.329770 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.330189 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.330193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.330196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.330396 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.331084 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:06.331088 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:06.331275 (+ 187us) client_negotiation.cc:769] Sending connection context
0619 04:46:06.332681 (+ 1406us) client_negotiation.cc:240] Negotiation successful
0619 04:46:06.333408 (+ 727us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":165,"spinlock_wait_cycles":35072,"thread_start_us":401,"threads_started":1}
I20260619 04:46:06.334843 11173 heartbeater.cc:347] Connected to a master server at 127.3.243.190:34723
I20260619 04:46:06.334949 11174 heartbeater.cc:347] Connected to a master server at 127.3.243.189:45177
I20260619 04:46:06.335207 11173 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:06.335278 11172 heartbeater.cc:347] Connected to a master server at 127.3.243.188:45313
I20260619 04:46:06.335389 11172 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:06.335212 11174 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:06.336227 11173 heartbeater.cc:511] Master 127.3.243.190:34723 requested a full tablet report, sending...
I20260619 04:46:06.336258 11172 heartbeater.cc:511] Master 127.3.243.188:45313 requested a full tablet report, sending...
I20260619 04:46:06.336504 11174 heartbeater.cc:511] Master 127.3.243.189:45177 requested a full tablet report, sending...
I20260619 04:46:06.338186 10204 ts_manager.cc:194] Re-registered known tserver with Master: 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301)
I20260619 04:46:06.338805 10803 ts_manager.cc:194] Registered new tserver with Master: 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301)
I20260619 04:46:06.338889 10993 ts_manager.cc:194] Registered new tserver with Master: 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301)
I20260619 04:46:06.339366 10204 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:35577
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.130@KRBTEST.COM
I20260619 04:46:06.349108 11182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.314068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.3.243.130:40913)
0619 04:46:06.326971 (+ 12903us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:06.326982 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:06.327010 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:46:06.327079 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:06.327088 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:06.327372 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:06.327535 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:06.331468 (+ 3933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.332968 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.335667 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.336576 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.340834 (+ 4258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.342444 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:06.342455 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:06.342467 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:06.342513 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:06.346160 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.346880 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.346898 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.346901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.347034 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.347424 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.347430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.347433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.347840 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:06.348181 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:06.348476 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 04:46:06.348736 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12709,"thread_start_us":13644,"threads_started":1}
I20260619 04:46:06.350302 10523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:06.433492 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 10400
I20260619 04:46:06.450114 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:40913
--local_ip_for_outbound_sockets=127.3.243.130
--tserver_master_addrs=127.3.243.190:34723
--webserver_port=34081
--webserver_interface=127.3.243.130
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab
--principal=oryx/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.243.190:34723,127.3.243.189:45177
--tserver_master_addrs=127.3.243.190:34723,127.3.243.189:45177,127.3.243.188:45313 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:46:06.542052 10951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: NON_VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: true } } }
I20260619 04:46:06.542462 10951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:06.544180 11185 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: attempting to promote NON_VOTER f80b1281aa2145729edb643968509225 to VOTER
I20260619 04:46:06.544834 11185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } }
I20260619 04:46:06.546772 10812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Refusing update from remote peer 7ec97feb168a47ad82c99a5bab5e6ed8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 04:46:06.546906 11003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 LEARNER]: Refusing update from remote peer 7ec97feb168a47ad82c99a5bab5e6ed8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 04:46:06.547435 11186 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 04:46:06.547857 10951 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 04:46:06.550278 10951 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f80b1281aa2145729edb643968509225 (127.3.243.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:06.552623 10812 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f80b1281aa2145729edb643968509225 (127.3.243.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:06.553435 10951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:06.553789 10951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:06.553599 11003 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f80b1281aa2145729edb643968509225 (127.3.243.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:06.554026 10951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:06.554252 10951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:06.554452 10951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:06.554656 10951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:06.557116 11032 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:06.557367 11032 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:46:06.559881 11189 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:06.560318 11189 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 04:46:06.783028 11184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:06.783303 11184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:06.783526 11184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:06.791606 11184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:06.791749 11184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:06.791922 11184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:46:06.801944 11184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab
--principal=oryx/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:40913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=34081
--tserver_master_addrs=127.3.243.190:34723,127.3.243.189:45177,127.3.243.188:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:06.803283 11184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:06.804700 11184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:06.818125 11194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:06.818089 11195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:06.818161 11197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:06.819205 11184 server_base.cc:1034] running on GCE node
I20260619 04:46:06.819854 11184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:06.820930 11184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:06.822152 11184 hybrid_clock.cc:648] HybridClock initialized: now 1781844366822103 us; error 45 us; skew 500 ppm
I20260619 04:46:06.822470 11184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844366, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:06.825889 11184 init.cc:377] Logged in from keytab as oryx/127.3.243.130@KRBTEST.COM (short username oryx)
I20260619 04:46:06.828058 11184 webserver.cc:460] Webserver started at http://127.3.243.130:34081/ using document root <none> and password file <none>
I20260619 04:46:06.828966 11184 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:06.829118 11184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:06.835804 11184 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 04:46:06.839063 11204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:06.840575 11184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 04:46:06.840719 11184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "91cb2bca3e284632bf8cd12eacfb2819"
format_stamp: "Formatted at 2026-06-19 04:46:03 on dist-test-slave-2qp8"
server_key: "79788dbc7043d1c992767e7fd1208dd8"
server_key_iv: "576bf7fa97a41cbd5cc44b2a449c3e3a"
server_key_version: "encryptionkey@0"
I20260619 04:46:06.841925 11184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:06.858893 11184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:06.863051 11184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:06.863284 11184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:06.864228 11184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:06.865501 11184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:06.865585 11184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:06.865687 11184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:06.865741 11184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:06.897893 11184 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:40913
I20260619 04:46:06.897985 11316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:40913 every 8 connection(s)
I20260619 04:46:06.899395 11184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:46:06.902163 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 11184
I20260619 04:46:06.902531 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 10533
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.131@KRBTEST.COM
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844366, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.188@KRBTEST.COM
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844366, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.189@KRBTEST.COM
Jun 19 04:46:06 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844366, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:06.947829 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.903937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.3.243.131:44163)
0619 04:46:06.918048 (+ 14111us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:06.918060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:06.918099 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:46:06.918186 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:06.918197 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:06.918511 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:06.918706 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:06.922035 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.923947 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.927641 (+ 3694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.928606 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.928724 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.935125 (+ 6401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:06.935142 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:06.935163 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:06.935206 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:06.941704 (+ 6498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.944163 (+ 2459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.944174 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.944180 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.944350 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.944834 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.944841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.944846 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.945389 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:06.945706 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:06.947116 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0619 04:46:06.947376 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13859,"thread_start_us":16556,"threads_started":1}
I20260619 04:46:06.947935 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.902236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:36175 (local address 127.3.243.189:45177)
0619 04:46:06.928039 (+ 25803us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:06.928046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:06.928065 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:46:06.928162 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:06.928168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:06.928312 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:06.929285 (+ 973us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:06.931104 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.931838 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.933996 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.934622 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.934735 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.940699 (+ 5964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:06.940710 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:06.940714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:06.940751 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:06.944264 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.945545 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.945550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.945554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.945694 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.946173 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.946180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.946185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.946568 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:06.946721 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:06.947308 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 04:46:06.947558 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25640,"thread_start_us":25496,"threads_started":1}
I20260619 04:46:06.948443 11324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.903259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:59279 (local address 127.3.243.188:45313)
0619 04:46:06.928257 (+ 24998us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:06.928263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:06.928282 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:46:06.928339 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:06.928343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:06.928424 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:06.928548 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:06.931144 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.932413 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.934424 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.935092 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.935200 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.940549 (+ 5349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:06.940555 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:06.940559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:06.940603 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:06.943627 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.944541 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.944549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.944554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.944696 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.945387 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.945392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.945395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.945752 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:06.945883 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:06.947707 (+ 1824us) server_negotiation.cc:299] Negotiation successful
0619 04:46:06.947966 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24822,"thread_start_us":24761,"threads_started":1}
I20260619 04:46:06.950068 10656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:06.950163 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.902831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.3.243.130:36175)
0619 04:46:06.914723 (+ 11892us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:06.914776 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:06.919622 (+ 4846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:06.928869 (+ 9247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:06.928881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:06.929809 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:06.930901 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.930926 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.931987 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.931992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.933855 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.933866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.935051 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.935058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.935375 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.936509 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:06.936563 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:06.940522 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:06.944427 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.944432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.944436 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.945389 (+ 953us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.945827 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.945832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.945836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.946034 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.946782 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:06.946788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:06.946938 (+ 150us) client_negotiation.cc:769] Sending connection context
0619 04:46:06.947067 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 04:46:06.949032 (+ 1965us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11596,"mutex_wait_us":65,"thread_start_us":11375,"threads_started":1}
I20260619 04:46:06.950186 11323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.902831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:45313 (local address 127.3.243.130:59279)
0619 04:46:06.926911 (+ 24080us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:06.926939 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:06.927087 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:06.928865 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:06.928879 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:06.930482 (+ 1603us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:06.930911 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.930928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.932604 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.932610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.934233 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.934247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.935464 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.935471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.936257 (+ 786us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.936923 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:06.936956 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:06.940275 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:06.943784 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.943844 (+ 60us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.943863 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.944382 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.944930 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.944935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.944940 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.945256 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.946068 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:06.946076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:06.946550 (+ 474us) client_negotiation.cc:769] Sending connection context
0619 04:46:06.946866 (+ 316us) client_negotiation.cc:240] Negotiation successful
0619 04:46:06.947389 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23859,"mutex_wait_us":68,"spinlock_wait_cycles":66944,"thread_start_us":23544,"threads_started":1}
I20260619 04:46:06.952006 11317 heartbeater.cc:347] Connected to a master server at 127.3.243.188:45313
I20260619 04:46:06.952046 11319 heartbeater.cc:347] Connected to a master server at 127.3.243.189:45177
I20260619 04:46:06.952370 11317 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:06.952378 11319 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:06.953372 11319 heartbeater.cc:511] Master 127.3.243.189:45177 requested a full tablet report, sending...
I20260619 04:46:06.953414 11317 heartbeater.cc:511] Master 127.3.243.188:45313 requested a full tablet report, sending...
I20260619 04:46:06.955135 10993 ts_manager.cc:194] Registered new tserver with Master: 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130:40913)
I20260619 04:46:06.955137 10803 ts_manager.cc:194] Registered new tserver with Master: 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130:40913)
I20260619 04:46:06.956166 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.915404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.130:38285)
0619 04:46:06.927275 (+ 11871us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:06.927298 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:06.927440 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:06.929691 (+ 2251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:06.929696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:06.930358 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:06.930901 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.930926 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.934121 (+ 3195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.934126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.936244 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:06.936261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.937990 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.937996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:06.938180 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.938732 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:06.938747 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:06.942475 (+ 3728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:06.948152 (+ 5677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.948157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.948161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.948672 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.949268 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:06.949273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:06.949276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:06.949464 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:06.954428 (+ 4964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:06.954435 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:06.954579 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 04:46:06.955478 (+ 899us) client_negotiation.cc:240] Negotiation successful
0619 04:46:06.955758 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11722,"mutex_wait_us":129,"thread_start_us":11580,"threads_started":1}
I20260619 04:46:06.956382 11328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:06.915565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:38285 (local address 127.3.243.190:34723)
0619 04:46:06.929358 (+ 13793us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:06.929364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:06.929385 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:46:06.929448 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:06.929456 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:06.929542 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:06.929654 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:06.932231 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.933256 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.937156 (+ 3900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:06.937777 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:06.937952 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:06.943158 (+ 5206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:06.943166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:06.943174 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:06.943222 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:06.947618 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.948805 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.948810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.948813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.948992 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:06.950572 (+ 1580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:06.950577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:06.950580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:06.950934 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:06.951068 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:06.955832 (+ 4764us) server_negotiation.cc:299] Negotiation successful
0619 04:46:06.956006 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13660,"thread_start_us":13561,"threads_started":1}
I20260619 04:46:06.957041 11318 heartbeater.cc:347] Connected to a master server at 127.3.243.190:34723
I20260619 04:46:06.957175 11318 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:06.957532 11318 heartbeater.cc:511] Master 127.3.243.190:34723 requested a full tablet report, sending...
I20260619 04:46:06.958427 10204 ts_manager.cc:194] Re-registered known tserver with Master: 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130:40913)
I20260619 04:46:06.959208 10204 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:38285
I20260619 04:46:07.038185 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 10533
I20260619 04:46:07.055478 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:44163
--local_ip_for_outbound_sockets=127.3.243.131
--tserver_master_addrs=127.3.243.190:34723
--webserver_port=42121
--webserver_interface=127.3.243.131
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab
--principal=oryx/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.243.190:34723,127.3.243.189:45177
--tserver_master_addrs=127.3.243.190:34723,127.3.243.189:45177,127.3.243.188:45313 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:46:07.342057 11173 heartbeater.cc:503] Master 127.3.243.190:34723 was elected leader, sending a full tablet report...
W20260619 04:46:07.384209 11330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:07.384538 11330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:07.384696 11330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:07.392518 11330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:07.392660 11330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:07.392827 11330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:46:07.402863 11330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab
--principal=oryx/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:44163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=42121
--tserver_master_addrs=127.3.243.190:34723,127.3.243.189:45177,127.3.243.188:45313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:07.404317 11330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:07.405706 11330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:07.419572 11338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:07.419543 11336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:07.419492 11335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:07.420477 11330 server_base.cc:1034] running on GCE node
I20260619 04:46:07.421090 11330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:07.421932 11330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:07.423133 11330 hybrid_clock.cc:648] HybridClock initialized: now 1781844367423095 us; error 66 us; skew 500 ppm
I20260619 04:46:07.423492 11330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:07 dist-test-slave-2qp8 krb5kdc[10160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844367, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:07.426683 11330 init.cc:377] Logged in from keytab as oryx/127.3.243.131@KRBTEST.COM (short username oryx)
I20260619 04:46:07.428047 11330 webserver.cc:460] Webserver started at http://127.3.243.131:42121/ using document root <none> and password file <none>
I20260619 04:46:07.428745 11330 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:07.428874 11330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:07.433959 11330 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 04:46:07.436517 11345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:07.437955 11330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 04:46:07.438113 11330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "e19e388de3464493a9e58d8ca26e8733"
format_stamp: "Formatted at 2026-06-19 04:46:04 on dist-test-slave-2qp8"
server_key: "04fbaad78ca65dca19ce8ec19ed5caa1"
server_key_iv: "9acb9458ec78278482c8a885daadf838"
server_key_version: "encryptionkey@0"
I20260619 04:46:07.439064 11330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:07.461056 11330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:07.465672 11330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:07.465978 11330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:07.467000 11330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:07.468374 11330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:07.468464 11330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:07.468544 11330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:07.468597 11330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:07.500754 11330 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:44163
I20260619 04:46:07.500813 11457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:44163 every 8 connection(s)
I20260619 04:46:07.502205 11330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:46:07.507290 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 11330
Jun 19 04:46:07 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844367, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.189@KRBTEST.COM
Jun 19 04:46:07 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844367, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.188@KRBTEST.COM
I20260619 04:46:07.533226 11466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.508404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:55203 (local address 127.3.243.189:45177)
0619 04:46:07.508871 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.508880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.509418 (+ 538us) server_negotiation.cc:407] Connection header received
0619 04:46:07.511394 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.511400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.511486 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.511600 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:07.513915 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.515345 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.517979 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.518901 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.519062 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.524458 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:07.524468 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:07.524476 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:07.524536 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:07.528482 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.529495 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.529500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.529504 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.529668 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.530163 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.530170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.530176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.530518 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:07.530663 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.532602 (+ 1939us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.532821 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":238,"threads_started":1}
I20260619 04:46:07.535209 11468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.508299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.3.243.131:55203)
0619 04:46:07.509324 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:07.509341 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:07.510564 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:07.511660 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:07.511666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:07.512939 (+ 1273us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:07.513734 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.513751 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.515496 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.515502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.517854 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.517864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.519150 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.519157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.519607 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.520444 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:07.520486 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:07.524206 (+ 3720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:07.528650 (+ 4444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:07.528661 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:07.528679 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:07.529350 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:07.529804 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:07.529809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:07.529813 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:07.530037 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:07.530735 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:07.530744 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:07.531176 (+ 432us) client_negotiation.cc:769] Sending connection context
0619 04:46:07.531475 (+ 299us) client_negotiation.cc:240] Negotiation successful
0619 04:46:07.532007 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":124,"thread_start_us":775,"threads_started":1}
I20260619 04:46:07.537477 11460 heartbeater.cc:347] Connected to a master server at 127.3.243.189:45177
I20260619 04:46:07.537806 11463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.505198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:58057 (local address 127.3.243.188:45313)
0619 04:46:07.506603 (+ 1405us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.506612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.508216 (+ 1604us) server_negotiation.cc:407] Connection header received
0619 04:46:07.510950 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.510954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.511057 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.511186 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:07.514609 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.515770 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.518027 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.518713 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.518838 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.527533 (+ 8695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:07.527544 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:07.527551 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:07.527612 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:07.532159 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.533494 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.533502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.533509 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.533703 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.535969 (+ 2266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.535974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.535978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.536376 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:07.536559 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.537226 (+ 667us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.537464 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1240,"thread_start_us":1117,"threads_started":1}
I20260619 04:46:07.537838 11460 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:07.538887 11460 heartbeater.cc:511] Master 127.3.243.189:45177 requested a full tablet report, sending...
Jun 19 04:46:07 dist-test-slave-2qp8 krb5kdc[10160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844367, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:07.539530 11464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.505730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:45313 (local address 127.3.243.131:58057)
0619 04:46:07.508100 (+ 2370us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:07.508138 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:07.510544 (+ 2406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:07.511783 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:07.511789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:07.514228 (+ 2439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:07.514461 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.514469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.515978 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.515983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.517850 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.517864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.519150 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.519157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.520029 (+ 872us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.521596 (+ 1567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:07.521620 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:07.527245 (+ 5625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:07.532382 (+ 5137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:07.532390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:07.532396 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:07.533042 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:07.535639 (+ 2597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:07.535644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:07.535647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:07.535830 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:07.536631 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:07.536637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:07.536802 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 04:46:07.536956 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 04:46:07.538286 (+ 1330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2176,"mutex_wait_us":896,"thread_start_us":717,"threads_started":1}
I20260619 04:46:07.540472 11458 heartbeater.cc:347] Connected to a master server at 127.3.243.188:45313
I20260619 04:46:07.540603 11458 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:07.540789 10803 ts_manager.cc:194] Registered new tserver with Master: e19e388de3464493a9e58d8ca26e8733 (127.3.243.131:44163)
I20260619 04:46:07.541908 11458 heartbeater.cc:511] Master 127.3.243.188:45313 requested a full tablet report, sending...
I20260619 04:46:07.543825 10993 ts_manager.cc:194] Registered new tserver with Master: e19e388de3464493a9e58d8ca26e8733 (127.3.243.131:44163)
I20260619 04:46:07.545310 11472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.511240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60060 (local address 127.3.243.190:34723)
0619 04:46:07.513568 (+ 2328us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.513576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.513598 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:46:07.513657 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.513665 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.513748 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.516080 (+ 2332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:07.519117 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.520409 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.522189 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.524534 (+ 2345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.524659 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.533593 (+ 8934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:07.533612 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:07.533619 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:07.533665 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:07.540346 (+ 6681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.541427 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.541433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.541436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.541615 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.542424 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.542431 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.542438 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.543101 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:07.544486 (+ 1385us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.544616 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.544874 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":2179,"thread_start_us":2077,"threads_started":1}
I20260619 04:46:07.550014 11465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.505730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.131:49273)
0619 04:46:07.507195 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:07.507239 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:07.510544 (+ 3305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:07.511591 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:07.511604 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:07.514849 (+ 3245us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:07.515071 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.515081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.517885 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.517894 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.521344 (+ 3450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.521363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.536249 (+ 14886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.536258 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.536463 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.537055 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:07.537077 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:07.542380 (+ 5303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:07.546558 (+ 4178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:07.546566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:07.546572 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:07.547044 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:07.547570 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:07.547575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:07.547578 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:07.547733 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:07.549010 (+ 1277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:07.549016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:07.549179 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 04:46:07.549393 (+ 214us) client_negotiation.cc:240] Negotiation successful
0619 04:46:07.549690 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":3,"thread_start_us":1876,"threads_started":1}
I20260619 04:46:07.550406 11462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.505165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:49273 (local address 127.3.243.190:34723)
0619 04:46:07.506154 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.506161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.507349 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 04:46:07.510872 (+ 3523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.510880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.510977 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.511063 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:07.515195 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.516314 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.521496 (+ 5182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.522095 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.522209 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.542597 (+ 20388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:07.542603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:07.542608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:07.542646 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:07.546241 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.547183 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.547188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.547191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.547328 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.547859 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.547865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.547869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.548262 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:07.548393 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.549493 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.549856 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":751,"threads_started":1}
I20260619 04:46:07.550590 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.513159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46076 (local address 127.3.243.189:45177)
0619 04:46:07.516576 (+ 3417us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.516581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.522237 (+ 5656us) server_negotiation.cc:407] Connection header received
0619 04:46:07.522297 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.522304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.522377 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.522495 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:07.528869 (+ 6374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.530114 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.531899 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.532518 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.532625 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.541176 (+ 8551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:07.541184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:07.541189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:07.541221 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:07.546527 (+ 5306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.547353 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.547360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.547365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.547589 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.548369 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.548374 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.548377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.548689 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:07.548945 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.549333 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.550185 (+ 852us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3277,"spinlock_wait_cycles":4480,"thread_start_us":3240,"threads_started":1}
I20260619 04:46:07.553213 11459 heartbeater.cc:347] Connected to a master server at 127.3.243.190:34723
I20260619 04:46:07.553371 11459 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:07.553869 11459 heartbeater.cc:511] Master 127.3.243.190:34723 requested a full tablet report, sending...
I20260619 04:46:07.554870 10205 ts_manager.cc:194] Re-registered known tserver with Master: e19e388de3464493a9e58d8ca26e8733 (127.3.243.131:44163)
I20260619 04:46:07.555768 10205 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:49273
I20260619 04:46:07.549944 10204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 04:46:07.559304 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.511757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57934 (local address 127.3.243.188:45313)
0619 04:46:07.512499 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.512507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.536101 (+ 23594us) server_negotiation.cc:407] Connection header received
0619 04:46:07.536369 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.536373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.536479 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.536564 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:07.541534 (+ 4970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.542998 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.545295 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.546016 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.546115 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.552616 (+ 6501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:07.552623 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:07.552628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:07.552665 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:07.556304 (+ 3639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.557112 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.557118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.557121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.557263 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.557677 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.557683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.557687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.558023 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:07.558284 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.558856 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.559028 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":448,"threads_started":1}
W20260619 04:46:07.560817 10204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:46:07.591008 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.580467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34301 (local address 127.0.0.1:55918)
0619 04:46:07.582438 (+ 1971us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:07.582458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:07.582654 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:07.583615 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:07.583620 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:07.583644 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:07.583935 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.583959 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.585864 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.585869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.588099 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.588132 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.590222 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.590228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.590415 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.590447 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:46:07.590529 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 04:46:07.590660 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1849,"threads_started":1}
I20260619 04:46:07.591970 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.581002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55918 (local address 127.3.243.129:34301)
0619 04:46:07.581552 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.581562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.582548 (+ 986us) server_negotiation.cc:407] Connection header received
0619 04:46:07.582851 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.582859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.583153 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.583334 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:07.584188 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.585679 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.588289 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.589713 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.589820 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.590040 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.591489 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.591669 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":650,"threads_started":1}
I20260619 04:46:07.601441 11107 tablet_service.cc:1467] Processing CreateTablet for tablet 021887ffda0640bca174e59087ce7005 (DEFAULT_TABLE table=test-table [id=062a8c3a1848414ca21c9c6eb0b2d9c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:07.603650 11107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021887ffda0640bca174e59087ce7005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:07.604264 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.581732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.3.243.130:40913)
0619 04:46:07.597955 (+ 16223us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.597964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.598004 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:46:07.598110 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.598119 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.598362 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.598573 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:07.598915 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.599853 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.602120 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.603237 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.603377 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.603591 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.603791 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.603954 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16045,"thread_start_us":16977,"threads_started":1}
I20260619 04:46:07.604593 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.581527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:40913 (local address 127.0.0.1:51080)
0619 04:46:07.591219 (+ 9692us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:07.591237 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:07.591326 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:07.598553 (+ 7227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:07.598557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:07.598583 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:07.598757 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.598768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.599997 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.600005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.601959 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.601970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.603388 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.603394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.603576 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.603611 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:46:07.603717 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 04:46:07.603979 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9596,"thread_start_us":677,"threads_started":1}
I20260619 04:46:07.608467 11252 tablet_service.cc:1467] Processing CreateTablet for tablet 021887ffda0640bca174e59087ce7005 (DEFAULT_TABLE table=test-table [id=062a8c3a1848414ca21c9c6eb0b2d9c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:07.609758 11252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021887ffda0640bca174e59087ce7005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:07.610463 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.596780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:44163 (local address 127.0.0.1:47876)
0619 04:46:07.597224 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:07.597244 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:07.597357 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:07.601208 (+ 3851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:07.601212 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:07.601237 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:07.602449 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.602462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.605360 (+ 2898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.605366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.607284 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.607299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.609653 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.609659 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.609903 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.609937 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:46:07.610043 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 04:46:07.610172 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307}
I20260619 04:46:07.610621 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.597047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.3.243.131:44163)
0619 04:46:07.600639 (+ 3592us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.600649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.600681 (+ 32us) server_negotiation.cc:407] Connection header received
0619 04:46:07.600771 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.600779 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.601007 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.601191 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:07.603714 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.605235 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.607476 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.609452 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.609586 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.609786 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.610113 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.610280 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3416,"thread_start_us":3242,"threads_started":1}
I20260619 04:46:07.614893 11393 tablet_service.cc:1467] Processing CreateTablet for tablet 021887ffda0640bca174e59087ce7005 (DEFAULT_TABLE table=test-table [id=062a8c3a1848414ca21c9c6eb0b2d9c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:07.617166 11393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021887ffda0640bca174e59087ce7005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:07.617775 11494 tablet_bootstrap.cc:492] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: Bootstrap starting.
I20260619 04:46:07.620587 11494 tablet_bootstrap.cc:654] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:07.621626 11494 log.cc:824] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:07.624853 11494 tablet_bootstrap.cc:492] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: No bootstrap required, opened a new log
I20260619 04:46:07.625108 11494 ts_tablet_manager.cc:1397] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 04:46:07.626538 11496 tablet_bootstrap.cc:492] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: Bootstrap starting.
I20260619 04:46:07.629886 11496 tablet_bootstrap.cc:654] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:07.631107 11496 log.cc:824] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:07.631588 11497 tablet_bootstrap.cc:492] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: Bootstrap starting.
I20260619 04:46:07.635176 11497 tablet_bootstrap.cc:654] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:07.635098 11494 raft_consensus.cc:348] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:07.635500 11496 tablet_bootstrap.cc:492] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: No bootstrap required, opened a new log
I20260619 04:46:07.635588 11494 raft_consensus.cc:374] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:07.635694 11494 raft_consensus.cc:729] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464f118ea75742279b82ad32617a1967, State: Initialized, Role: FOLLOWER
I20260619 04:46:07.635769 11496 ts_tablet_manager.cc:1397] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260619 04:46:07.636343 11494 consensus_queue.cc:260] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:07.636987 11497 log.cc:824] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:07.638991 11494 ts_tablet_manager.cc:1428] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260619 04:46:07.641065 11497 tablet_bootstrap.cc:492] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: No bootstrap required, opened a new log
I20260619 04:46:07.641327 11497 ts_tablet_manager.cc:1397] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260619 04:46:07.648741 11496 raft_consensus.cc:348] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:07.649089 11496 raft_consensus.cc:374] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:07.649195 11496 raft_consensus.cc:729] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91cb2bca3e284632bf8cd12eacfb2819, State: Initialized, Role: FOLLOWER
I20260619 04:46:07.649734 11496 consensus_queue.cc:260] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:07.649962 11497 raft_consensus.cc:348] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:07.650394 11497 raft_consensus.cc:374] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:07.650504 11497 raft_consensus.cc:729] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19e388de3464493a9e58d8ca26e8733, State: Initialized, Role: FOLLOWER
I20260619 04:46:07.650792 11318 heartbeater.cc:503] Master 127.3.243.190:34723 was elected leader, sending a full tablet report...
I20260619 04:46:07.651085 11497 consensus_queue.cc:260] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:07.651330 11496 ts_tablet_manager.cc:1428] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260619 04:46:07.652817 11497 ts_tablet_manager.cc:1428] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260619 04:46:07.653096 11459 heartbeater.cc:503] Master 127.3.243.190:34723 was elected leader, sending a full tablet report...
W20260619 04:46:07.658752 11320 tablet.cc:2369] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:46:07.683420 11501 raft_consensus.cc:482] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:07.683780 11501 raft_consensus.cc:504] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:07.685504 11501 leader_election.cc:288] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301), e19e388de3464493a9e58d8ca26e8733 (127.3.243.131:44163)
I20260619 04:46:07.692585 11490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.685865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:50007 (local address 127.3.243.129:34301)
0619 04:46:07.686105 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.686113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.686461 (+ 348us) server_negotiation.cc:407] Connection header received
0619 04:46:07.686696 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.686701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.686779 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.686845 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:07.687546 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.688614 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.690616 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.691767 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.691885 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.691947 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.692305 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.692402 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:46:07.692821 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.685718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34301 (local address 127.3.243.130:50007)
0619 04:46:07.686318 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:07.686333 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:07.686505 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:07.687052 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:07.687055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:07.687080 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:07.687387 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.687395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.688735 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.688740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.690461 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.690472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.691951 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.691957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.692151 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.692187 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:46:07.692284 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 04:46:07.692425 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":1,"thread_start_us":313,"threads_started":1}
I20260619 04:46:07.693931 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.686390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:44163 (local address 127.3.243.130:41227)
0619 04:46:07.687418 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:07.687433 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:07.687648 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:07.687995 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:07.687999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:07.688024 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:07.688309 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.688320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.689699 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.689702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.691389 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:07.691403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.693175 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.693182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:07.693390 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.693424 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:46:07.693500 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 04:46:07.693588 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":1,"thread_start_us":746,"threads_started":1}
I20260619 04:46:07.693958 11493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.686647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:41227 (local address 127.3.243.131:44163)
0619 04:46:07.686843 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.686849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.687542 (+ 693us) server_negotiation.cc:407] Connection header received
0619 04:46:07.687820 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.687824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.687882 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.687951 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:07.688467 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.689564 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.691546 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.692685 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.692852 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.692930 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.693560 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.693718 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:46:07.693954 11127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021887ffda0640bca174e59087ce7005" candidate_uuid: "91cb2bca3e284632bf8cd12eacfb2819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464f118ea75742279b82ad32617a1967" is_pre_election: true
I20260619 04:46:07.694581 11127 raft_consensus.cc:2454] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91cb2bca3e284632bf8cd12eacfb2819 in term 0.
I20260619 04:46:07.695009 11413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021887ffda0640bca174e59087ce7005" candidate_uuid: "91cb2bca3e284632bf8cd12eacfb2819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19e388de3464493a9e58d8ca26e8733" is_pre_election: true
I20260619 04:46:07.695259 11207 leader_election.cc:302] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 464f118ea75742279b82ad32617a1967, 91cb2bca3e284632bf8cd12eacfb2819; no voters:
I20260619 04:46:07.695513 11413 raft_consensus.cc:2454] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91cb2bca3e284632bf8cd12eacfb2819 in term 0.
I20260619 04:46:07.696079 11501 raft_consensus.cc:2781] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:46:07.696271 11501 raft_consensus.cc:482] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:46:07.696389 11501 raft_consensus.cc:3037] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:07.698576 11501 raft_consensus.cc:504] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:07.699155 11501 leader_election.cc:288] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [CANDIDATE]: Term 1 election: Requested vote from peers 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301), e19e388de3464493a9e58d8ca26e8733 (127.3.243.131:44163)
I20260619 04:46:07.699522 11127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021887ffda0640bca174e59087ce7005" candidate_uuid: "91cb2bca3e284632bf8cd12eacfb2819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464f118ea75742279b82ad32617a1967"
I20260619 04:46:07.699633 11413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021887ffda0640bca174e59087ce7005" candidate_uuid: "91cb2bca3e284632bf8cd12eacfb2819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19e388de3464493a9e58d8ca26e8733"
I20260619 04:46:07.699714 11127 raft_consensus.cc:3037] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:07.699864 11413 raft_consensus.cc:3037] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:07.701886 11127 raft_consensus.cc:2454] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91cb2bca3e284632bf8cd12eacfb2819 in term 1.
I20260619 04:46:07.702088 11413 raft_consensus.cc:2454] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91cb2bca3e284632bf8cd12eacfb2819 in term 1.
I20260619 04:46:07.702308 11207 leader_election.cc:302] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 464f118ea75742279b82ad32617a1967, 91cb2bca3e284632bf8cd12eacfb2819; no voters:
I20260619 04:46:07.702598 11501 raft_consensus.cc:2781] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:07.703030 11501 raft_consensus.cc:686] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 1 LEADER]: Becoming Leader. State: Replica: 91cb2bca3e284632bf8cd12eacfb2819, State: Running, Role: LEADER
I20260619 04:46:07.703612 11501 consensus_queue.cc:237] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:07.708758 10205 catalog_manager.cc:5495] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130). New cstate: current_term: 1 leader_uuid: "91cb2bca3e284632bf8cd12eacfb2819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } health_report { overall_health: HEALTHY } } }
I20260619 04:46:07.750536 11492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.745331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.3.243.130:40913)
0619 04:46:07.745496 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.745501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.745519 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:46:07.745624 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.745627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.745705 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.745822 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:46:07.746203 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.746914 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.748380 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.748792 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.748931 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.749414 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:46:07.749916 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:46:07.750104 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.750163 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.750243 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 04:46:07.755057 11461 tablet.cc:2369] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:46:07.758862 11413 raft_consensus.cc:1261] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [term 1 FOLLOWER]: Refusing update from remote peer 91cb2bca3e284632bf8cd12eacfb2819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:07.758891 11127 raft_consensus.cc:1261] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [term 1 FOLLOWER]: Refusing update from remote peer 91cb2bca3e284632bf8cd12eacfb2819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:07.759882 11505 consensus_queue.cc:1035] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:07.760299 11501 consensus_queue.cc:1035] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:07.772537 11508 mvcc.cc:205] Tried to move back new op lower bound from 7298434530331045888 to 7298434530117849088. Current Snapshot: MvccSnapshot[applied={T|T < 7298434530331045888}]
I20260619 04:46:07.779068 11509 mvcc.cc:205] Tried to move back new op lower bound from 7298434530331045888 to 7298434530117849088. Current Snapshot: MvccSnapshot[applied={T|T < 7298434530331045888}]
I20260619 04:46:07.800408 10204 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:07.800679 10204 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:07.806869 10204 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ec97feb168a47ad82c99a5bab5e6ed8: Sending DeleteTablet for 3 replicas of tablet 021887ffda0640bca174e59087ce7005
I20260619 04:46:07.808351 11393 tablet_service.cc:1514] Processing DeleteTablet for tablet 021887ffda0640bca174e59087ce7005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:07 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:47876
I20260619 04:46:07.809453 11107 tablet_service.cc:1514] Processing DeleteTablet for tablet 021887ffda0640bca174e59087ce7005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:07 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:55918
I20260619 04:46:07.809983 11252 tablet_service.cc:1514] Processing DeleteTablet for tablet 021887ffda0640bca174e59087ce7005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:07 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:51080
I20260619 04:46:07.818234 11517 tablet_replica.cc:330] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: stopping tablet replica
I20260619 04:46:07.819121 11517 raft_consensus.cc:2229] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:07.820569 11517 raft_consensus.cc:2258] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:07.823794 11517 ts_tablet_manager.cc:1905] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:07.825737 11518 tablet_replica.cc:330] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: stopping tablet replica
I20260619 04:46:07.826298 11518 raft_consensus.cc:2229] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:07.827124 11518 raft_consensus.cc:2258] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:07.829941 11518 ts_tablet_manager.cc:1905] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:07.837984 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 10173
I20260619 04:46:07.839393 10235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:07.845674 11518 ts_tablet_manager.cc:1918] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:07.845901 11518 log.cc:1196] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/021887ffda0640bca174e59087ce7005
I20260619 04:46:07.846571 11518 ts_tablet_manager.cc:1939] T 021887ffda0640bca174e59087ce7005 P 464f118ea75742279b82ad32617a1967: Deleting consensus metadata
I20260619 04:46:07.851568 10191 catalog_manager.cc:4841] TS 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301): tablet 021887ffda0640bca174e59087ce7005 (table test-table [id=062a8c3a1848414ca21c9c6eb0b2d9c3]) successfully deleted
I20260619 04:46:07.868693 11519 tablet_replica.cc:330] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: stopping tablet replica
I20260619 04:46:07.869307 11519 raft_consensus.cc:2229] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:46:07.870144 11519 raft_consensus.cc:2258] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:07.872493 11517 ts_tablet_manager.cc:1918] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:07.872722 11517 log.cc:1196] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/021887ffda0640bca174e59087ce7005
I20260619 04:46:07.873401 11517 ts_tablet_manager.cc:1939] T 021887ffda0640bca174e59087ce7005 P e19e388de3464493a9e58d8ca26e8733: Deleting consensus metadata
I20260619 04:46:07.877764 11519 ts_tablet_manager.cc:1905] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:07.878953 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.811967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46080 (local address 127.3.243.189:45177)
0619 04:46:07.812224 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.812233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.825325 (+ 13092us) server_negotiation.cc:407] Connection header received
0619 04:46:07.842525 (+ 17200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.842532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.842625 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.842725 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:07.851922 (+ 9197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.853636 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.861125 (+ 7489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.861859 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.861972 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.871383 (+ 9411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:07.871391 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:07.871396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:07.871458 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:07.875150 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.876242 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.876248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.876252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.876433 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.876910 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.876917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.876920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.877304 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:07.877609 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.878346 (+ 737us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.878595 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:46:07.886919 11519 ts_tablet_manager.cc:1918] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:07.887106 11519 log.cc:1196] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/021887ffda0640bca174e59087ce7005
I20260619 04:46:07.888078 11519 ts_tablet_manager.cc:1939] T 021887ffda0640bca174e59087ce7005 P 91cb2bca3e284632bf8cd12eacfb2819: Deleting consensus metadata
I20260619 04:46:07.901014 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:07.826440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.3.243.188:45313)
0619 04:46:07.832294 (+ 5854us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:07.832300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:07.880635 (+ 48335us) server_negotiation.cc:407] Connection header received
0619 04:46:07.880874 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:07.880879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:07.880991 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:07.881060 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:07.883166 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.884392 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.888689 (+ 4297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:07.889382 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:07.889484 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:07.892532 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:07.892538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:07.892541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:07.892575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:07.895454 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.896085 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.896091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.896095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.896263 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:07.896660 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:07.896665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:07.896669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:07.896946 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:07.900302 (+ 3356us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:07.900443 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 04:46:07.900679 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5689}
I20260619 04:46:08.044279 10191 catalog_manager.cc:4841] TS e19e388de3464493a9e58d8ca26e8733 (127.3.243.131:44163): tablet 021887ffda0640bca174e59087ce7005 (table test-table [id=062a8c3a1848414ca21c9c6eb0b2d9c3]) successfully deleted
I20260619 04:46:08.044431 10192 catalog_manager.cc:4841] TS 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130:40913): tablet 021887ffda0640bca174e59087ce7005 (table test-table [id=062a8c3a1848414ca21c9c6eb0b2d9c3]) successfully deleted
I20260619 04:46:08.044889 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 10173
I20260619 04:46:08.088760 11475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:08.072566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.3.243.189:45177)
0619 04:46:08.072786 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:08.072793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:08.073477 (+ 684us) server_negotiation.cc:407] Connection header received
0619 04:46:08.073635 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:08.073640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:08.073717 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:08.073808 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:08.075788 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:08.077030 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:08.079115 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:08.079656 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:08.079784 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:08.082448 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:08.082459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:08.082463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:08.082493 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:08.085662 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:08.086273 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:08.086278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:08.086282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:08.086407 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:08.086775 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:08.086798 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:08.086802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:08.087179 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:08.087443 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:08.088084 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 04:46:08.088397 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:46:08.089594 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:08.071514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.3.243.188:45313)
0619 04:46:08.071750 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:08.071759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:08.073218 (+ 1459us) server_negotiation.cc:407] Connection header received
0619 04:46:08.073587 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:08.073596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:08.073687 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:08.073796 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:08.075995 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:08.077110 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:08.078611 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:08.079183 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:08.079301 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:08.082999 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:08.083007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:08.083014 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:08.083063 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:08.086559 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:08.087262 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:08.087267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:08.087271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:08.087409 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:08.087763 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:08.087769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:08.087772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:08.088209 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:08.088440 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:08.089012 (+ 572us) server_negotiation.cc:299] Negotiation successful
0619 04:46:08.089263 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:46:08.785965 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:08.785215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.131:52053)
0619 04:46:08.785658 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:08.785780 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":217,"threads_started":1}
W20260619 04:46:08.786382 11459 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:34723 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
I20260619 04:46:08.789661 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:08.788744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.130:46251)
0619 04:46:08.789291 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:08.789443 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":295,"threads_started":1}
W20260619 04:46:08.790403 11318 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:34723 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
I20260619 04:46:08.794179 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:08.793464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.129:33099)
0619 04:46:08.793864 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:08.793979 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":171,"threads_started":1}
W20260619 04:46:08.794633 11173 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:34723 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
I20260619 04:46:09.306965 11532 raft_consensus.cc:482] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ec97feb168a47ad82c99a5bab5e6ed8)
I20260619 04:46:09.307219 11532 raft_consensus.cc:504] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } }
I20260619 04:46:09.308557 11533 raft_consensus.cc:482] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ec97feb168a47ad82c99a5bab5e6ed8)
I20260619 04:46:09.308612 11532 leader_election.cc:288] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ec97feb168a47ad82c99a5bab5e6ed8 (127.3.243.190:34723), b318ae68e78d4f35ab54cfdfa7248430 (127.3.243.189:45177)
I20260619 04:46:09.308882 11533 raft_consensus.cc:504] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } }
I20260619 04:46:09.309747 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.308832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60500)
0619 04:46:09.309285 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.309467 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":199,"threads_started":1}
W20260619 04:46:09.310985 10980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
I20260619 04:46:09.311297 11533 leader_election.cc:288] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ec97feb168a47ad82c99a5bab5e6ed8 (127.3.243.190:34723), f80b1281aa2145729edb643968509225 (127.3.243.188:45313)
I20260619 04:46:09.311772 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.310618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60510)
0619 04:46:09.311207 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.311394 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":302,"threads_started":1}
W20260619 04:46:09.312151 10790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
I20260619 04:46:09.314083 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.313458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60512)
0619 04:46:09.313741 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.313877 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260619 04:46:09.314324 10790 leader_election.cc:334] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ec97feb168a47ad82c99a5bab5e6ed8 (127.3.243.190:34723): Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
I20260619 04:46:09.314405 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.313831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60522)
0619 04:46:09.313979 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.314143 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260619 04:46:09.314687 10980 leader_election.cc:334] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ec97feb168a47ad82c99a5bab5e6ed8 (127.3.243.190:34723): Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
I20260619 04:46:09.317400 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.309452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:45177 (local address 127.0.0.1:46112)
0619 04:46:09.309977 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.309997 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:09.310163 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:09.311144 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:09.311149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:09.311181 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:09.311419 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.311434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.313373 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.313378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.315378 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.315391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.316604 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.316610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.316794 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.316818 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 04:46:09.316946 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 04:46:09.317086 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":1,"thread_start_us":1201,"threads_started":1}
I20260619 04:46:09.317549 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.309551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.3.243.189:45177)
0619 04:46:09.310754 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.310761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.310783 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:46:09.310861 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.310869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.310964 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.311593 (+ 629us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:09.311848 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.313180 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.315556 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.316448 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.316598 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.316697 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.316975 (+ 278us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.317111 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1072,"thread_start_us":872,"threads_started":1}
I20260619 04:46:09.318176 10812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f80b1281aa2145729edb643968509225" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b318ae68e78d4f35ab54cfdfa7248430" is_pre_election: true
I20260619 04:46:09.318480 10812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f80b1281aa2145729edb643968509225 in term 1.
I20260619 04:46:09.318957 10980 leader_election.cc:302] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b318ae68e78d4f35ab54cfdfa7248430, f80b1281aa2145729edb643968509225; no voters: 7ec97feb168a47ad82c99a5bab5e6ed8
I20260619 04:46:09.319307 11532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 04:46:09.319412 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.311683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:45313 (local address 127.0.0.1:57966)
0619 04:46:09.312126 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.312145 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:09.312255 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:09.314179 (+ 1924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:09.314186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:09.314221 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:09.314518 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.314531 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.315907 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.315912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.317313 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.317325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.318664 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.318670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.318837 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.318875 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 04:46:09.318975 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 04:46:09.319108 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":1512,"threads_started":1}
I20260619 04:46:09.319449 11532 raft_consensus.cc:482] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7ec97feb168a47ad82c99a5bab5e6ed8)
I20260619 04:46:09.319536 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.312197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.3.243.188:45313)
0619 04:46:09.313835 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.313840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.313861 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:46:09.313914 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.313919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.314004 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.314662 (+ 658us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:09.314954 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.315772 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.317461 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.318481 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.318616 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.318728 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.318998 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.319118 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1522,"thread_start_us":1366,"threads_started":1}
I20260619 04:46:09.319557 11532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:46:09.320025 11003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b318ae68e78d4f35ab54cfdfa7248430" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f80b1281aa2145729edb643968509225" is_pre_election: true
I20260619 04:46:09.322451 11532 raft_consensus.cc:504] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } }
I20260619 04:46:09.322880 11003 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b318ae68e78d4f35ab54cfdfa7248430 in current term 2: Already voted for candidate f80b1281aa2145729edb643968509225 in this term.
I20260619 04:46:09.323078 11532 leader_election.cc:288] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [CANDIDATE]: Term 2 election: Requested vote from peers 7ec97feb168a47ad82c99a5bab5e6ed8 (127.3.243.190:34723), b318ae68e78d4f35ab54cfdfa7248430 (127.3.243.189:45177)
I20260619 04:46:09.323625 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.323192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60546)
0619 04:46:09.323329 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.323418 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 04:46:09.323706 10812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f80b1281aa2145729edb643968509225" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b318ae68e78d4f35ab54cfdfa7248430"
I20260619 04:46:09.323967 10812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:46:09.323942 10789 leader_election.cc:302] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b318ae68e78d4f35ab54cfdfa7248430; no voters: 7ec97feb168a47ad82c99a5bab5e6ed8, f80b1281aa2145729edb643968509225
I20260619 04:46:09.324959 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.324607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60562)
0619 04:46:09.324763 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.324843 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 04:46:09.325117 10980 leader_election.cc:334] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7ec97feb168a47ad82c99a5bab5e6ed8 (127.3.243.190:34723): Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
I20260619 04:46:09.326462 10812 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f80b1281aa2145729edb643968509225 in term 2.
I20260619 04:46:09.326622 11533 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:46:09.326838 10980 leader_election.cc:302] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b318ae68e78d4f35ab54cfdfa7248430, f80b1281aa2145729edb643968509225; no voters: 7ec97feb168a47ad82c99a5bab5e6ed8
I20260619 04:46:09.327102 11532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:46:09.327456 11532 raft_consensus.cc:686] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [term 2 LEADER]: Becoming Leader. State: Replica: f80b1281aa2145729edb643968509225, State: Running, Role: LEADER
I20260619 04:46:09.327793 11532 consensus_queue.cc:237] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } }
I20260619 04:46:09.329547 11542 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f80b1281aa2145729edb643968509225. Latest consensus state: current_term: 2 leader_uuid: "f80b1281aa2145729edb643968509225" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:09.329988 11542 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:09.330581 11544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:46:09.333966 11544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:46:09.334700 11544 catalog_manager.cc:1259] Loaded cluster ID: 65a5930847eb4d36b4f0872cb67076f5
I20260619 04:46:09.334793 11544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:46:09.335906 11544 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:46:09.336740 11544 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Loaded TSK: 0
I20260619 04:46:09.337534 11544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:46:09.383343 10993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:46:09.390333 10993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:46:09.394819 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.394303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60570)
0619 04:46:09.394520 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.394616 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 04:46:09.394783 10812 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [term 2 FOLLOWER]: Refusing update from remote peer f80b1281aa2145729edb643968509225: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 04:46:09.395583 11532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 04:46:09.396034 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.395656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60582)
0619 04:46:09.395801 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.395880 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 04:46:09.396348 10980 consensus_peers.cc:487] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 -> Peer 7ec97feb168a47ad82c99a5bab5e6ed8 (127.3.243.190:34723): Couldn't send request to peer 7ec97feb168a47ad82c99a5bab5e6ed8. Status: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 04:46:09.398303 11533 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f80b1281aa2145729edb643968509225. Latest consensus state: current_term: 2 leader_uuid: "f80b1281aa2145729edb643968509225" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:09.398577 11533 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:46:09.399981 11542 sys_catalog.cc:455] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f80b1281aa2145729edb643968509225" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:09.400271 11542 sys_catalog.cc:458] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:09.400486 11533 sys_catalog.cc:455] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f80b1281aa2145729edb643968509225" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7ec97feb168a47ad82c99a5bab5e6ed8" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 34723 } } peers { permanent_uuid: "b318ae68e78d4f35ab54cfdfa7248430" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 45177 } attrs { promote: false } } peers { permanent_uuid: "f80b1281aa2145729edb643968509225" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 45313 } attrs { promote: false } } }
I20260619 04:46:09.400780 11533 sys_catalog.cc:458] T 00000000000000000000000000000000 P b318ae68e78d4f35ab54cfdfa7248430 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:46:09.416208 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.409163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.3.243.129:34301)
0619 04:46:09.409622 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.409627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.409644 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:46:09.409713 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.409720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.409798 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.409905 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:09.410386 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.411752 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.414195 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.414969 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.415129 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.415244 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.415519 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.415661 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":241,"threads_started":1}
I20260619 04:46:09.416831 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.408443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:44163 (local address 127.0.0.1:47888)
0619 04:46:09.408625 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.408659 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:09.408787 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:09.409614 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:09.409619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:09.409643 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:09.409827 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.409836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.411304 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.411312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.413570 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.413601 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.415514 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.415537 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.415825 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.415882 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 04:46:09.415959 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 04:46:09.416174 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 04:46:09.417109 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.408833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34301 (local address 127.0.0.1:55928)
0619 04:46:09.409004 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.409023 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:09.409126 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:09.409930 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:09.409933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:09.409960 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:09.410248 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.410258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.411965 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.411970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.414057 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.414069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.415180 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.415187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.415362 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.415391 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 04:46:09.415494 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 04:46:09.415637 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 04:46:09.417326 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.408569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47888 (local address 127.3.243.131:44163)
0619 04:46:09.409207 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.409214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.409232 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:46:09.409307 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.409315 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.409429 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.409598 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:09.410000 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.411104 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.413790 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.415305 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.416597 (+ 1292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.416742 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.416795 (+ 53us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.416933 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":411,"threads_started":1}
I20260619 04:46:09.417793 11107 tablet_service.cc:1467] Processing CreateTablet for tablet 42be192df15847c684cc5e5ad80a4b1c (DEFAULT_TABLE table=test-table [id=b2805cbe9fa8484f9f2c7c483df8b468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:09.418263 11107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be192df15847c684cc5e5ad80a4b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:09.418380 11393 tablet_service.cc:1467] Processing CreateTablet for tablet 42be192df15847c684cc5e5ad80a4b1c (DEFAULT_TABLE table=test-table [id=b2805cbe9fa8484f9f2c7c483df8b468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:09.418920 11393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be192df15847c684cc5e5ad80a4b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:09.423895 11555 tablet_bootstrap.cc:492] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: Bootstrap starting.
I20260619 04:46:09.425412 11555 tablet_bootstrap.cc:654] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:09.427683 11555 tablet_bootstrap.cc:492] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: No bootstrap required, opened a new log
I20260619 04:46:09.427837 11555 ts_tablet_manager.cc:1397] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 04:46:09.429392 11557 tablet_bootstrap.cc:492] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: Bootstrap starting.
I20260619 04:46:09.428663 11555 raft_consensus.cc:348] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.430864 11555 raft_consensus.cc:374] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:09.430977 11555 raft_consensus.cc:729] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464f118ea75742279b82ad32617a1967, State: Initialized, Role: FOLLOWER
I20260619 04:46:09.431126 11557 tablet_bootstrap.cc:654] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:09.431344 11555 consensus_queue.cc:260] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.432548 11555 ts_tablet_manager.cc:1428] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260619 04:46:09.435290 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.410690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:40913 (local address 127.0.0.1:51106)
0619 04:46:09.412730 (+ 2040us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.412747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:09.412854 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:09.428824 (+ 15970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:09.428829 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:09.428856 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:09.429252 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.429269 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.430687 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.430692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.432710 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.432730 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.434444 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.434451 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.434645 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.434678 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:46:09.434780 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 04:46:09.434930 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1906,"thread_start_us":1505,"threads_started":1}
I20260619 04:46:09.436101 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.410784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.3.243.130:40913)
0619 04:46:09.428479 (+ 17695us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.428486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.428506 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:46:09.428575 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.428580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.428683 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.428790 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:09.429469 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.430563 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.432910 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.434066 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.434188 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.434292 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.435627 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.435773 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17572,"thread_start_us":17478,"threads_started":1}
I20260619 04:46:09.436880 11252 tablet_service.cc:1467] Processing CreateTablet for tablet 42be192df15847c684cc5e5ad80a4b1c (DEFAULT_TABLE table=test-table [id=b2805cbe9fa8484f9f2c7c483df8b468]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:09.437357 11252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42be192df15847c684cc5e5ad80a4b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:09.437958 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.433306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.129:33889)
0619 04:46:09.437557 (+ 4251us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.437704 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4120,"thread_start_us":3969,"threads_started":1}
I20260619 04:46:09.438352 11557 tablet_bootstrap.cc:492] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: No bootstrap required, opened a new log
I20260619 04:46:09.438493 11557 ts_tablet_manager.cc:1397] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260619 04:46:09.439343 11557 raft_consensus.cc:348] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.439582 11557 raft_consensus.cc:374] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:09.439677 11557 raft_consensus.cc:729] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e19e388de3464493a9e58d8ca26e8733, State: Initialized, Role: FOLLOWER
I20260619 04:46:09.439893 11557 consensus_queue.cc:260] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.440665 11557 ts_tablet_manager.cc:1428] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:46:09.443197 11458 heartbeater.cc:503] Master 127.3.243.188:45313 was elected leader, sending a full tablet report...
I20260619 04:46:09.443472 11563 tablet_bootstrap.cc:492] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: Bootstrap starting.
I20260619 04:46:09.445361 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.440905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.131:45253)
0619 04:46:09.444716 (+ 3811us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.444845 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3649,"thread_start_us":3438,"threads_started":1}
I20260619 04:46:09.445425 11563 tablet_bootstrap.cc:654] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:09.446689 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.446255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.131:60643)
0619 04:46:09.446459 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.446557 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 04:46:09.447889 11563 tablet_bootstrap.cc:492] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: No bootstrap required, opened a new log
I20260619 04:46:09.448040 11563 ts_tablet_manager.cc:1397] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 04:46:09.448676 11563 raft_consensus.cc:348] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.448845 11563 raft_consensus.cc:374] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:09.448911 11563 raft_consensus.cc:729] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91cb2bca3e284632bf8cd12eacfb2819, State: Initialized, Role: FOLLOWER
I20260619 04:46:09.449118 11563 consensus_queue.cc:260] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.449790 11563 ts_tablet_manager.cc:1428] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 04:46:09.451131 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.450112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.130:54605)
0619 04:46:09.450710 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.450843 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":350,"threads_started":1}
I20260619 04:46:09.452090 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.451733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.130:43137)
0619 04:46:09.451852 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.451952 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 04:46:09.452098 11317 heartbeater.cc:503] Master 127.3.243.188:45313 was elected leader, sending a full tablet report...
W20260619 04:46:09.560669 11175 tablet.cc:2369] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:46:09.666155 11561 raft_consensus.cc:482] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:09.666385 11561 raft_consensus.cc:504] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.667891 11558 raft_consensus.cc:482] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:09.668028 11561 leader_election.cc:288] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301), 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130:40913)
I20260619 04:46:09.668166 11558 raft_consensus.cc:504] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.670006 11558 leader_election.cc:288] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e19e388de3464493a9e58d8ca26e8733 (127.3.243.131:44163), 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130:40913)
I20260619 04:46:09.676296 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.668725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:40913 (local address 127.3.243.131:43141)
0619 04:46:09.669325 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.669346 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:09.669455 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:09.669819 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:09.669823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:09.669850 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:09.670104 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.670116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.671457 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.671462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.673496 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.673509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.675371 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.675379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.675606 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.675644 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 04:46:09.675729 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 04:46:09.675904 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":345,"threads_started":1}
I20260619 04:46:09.676681 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.668510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:51483 (local address 127.3.243.129:34301)
0619 04:46:09.668755 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.668778 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.668799 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:46:09.668869 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.668873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.668955 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.669062 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:09.669706 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.670875 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.673571 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.675521 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.675676 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.675742 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.676278 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.676400 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":74}
I20260619 04:46:09.676935 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.668526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:43141 (local address 127.3.243.130:40913)
0619 04:46:09.669051 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.669057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.669439 (+ 382us) server_negotiation.cc:407] Connection header received
0619 04:46:09.669607 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.669612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.669688 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.669791 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:09.670283 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.671302 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.673649 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.674997 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.675140 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.675220 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.676516 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.676666 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368}
I20260619 04:46:09.677883 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.668235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34301 (local address 127.3.243.131:51483)
0619 04:46:09.668438 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.668458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:09.668603 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:09.669243 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:09.669250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:09.669275 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:09.669569 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.669581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.671106 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.671111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.673352 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.673368 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.675884 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.675890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.676072 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.676102 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:46:09.677432 (+ 1330us) client_negotiation.cc:240] Negotiation successful
0619 04:46:09.677594 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 04:46:09.677799 11272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be192df15847c684cc5e5ad80a4b1c" candidate_uuid: "e19e388de3464493a9e58d8ca26e8733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91cb2bca3e284632bf8cd12eacfb2819" is_pre_election: true
I20260619 04:46:09.678256 11272 raft_consensus.cc:2454] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e19e388de3464493a9e58d8ca26e8733 in term 0.
I20260619 04:46:09.678455 11127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be192df15847c684cc5e5ad80a4b1c" candidate_uuid: "e19e388de3464493a9e58d8ca26e8733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464f118ea75742279b82ad32617a1967" is_pre_election: true
I20260619 04:46:09.678723 11127 raft_consensus.cc:2454] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e19e388de3464493a9e58d8ca26e8733 in term 0.
I20260619 04:46:09.679239 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.670191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:44163 (local address 127.3.243.129:53293)
0619 04:46:09.670478 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.670510 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:09.670625 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:09.671382 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:09.671389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:09.671424 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:09.671771 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.671784 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.673677 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.673685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.676624 (+ 2939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.676637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.678447 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.678454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.678643 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.678675 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:46:09.678776 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 04:46:09.678922 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":280}
I20260619 04:46:09.679247 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.670824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:53293 (local address 127.3.243.131:44163)
0619 04:46:09.671024 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.671033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.671054 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:46:09.671118 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.671125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.671215 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.671925 (+ 710us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:09.672109 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.673505 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.676802 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.678070 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.678194 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.678259 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.678860 (+ 601us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.678991 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":61}
I20260619 04:46:09.679171 11349 leader_election.cc:302] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91cb2bca3e284632bf8cd12eacfb2819, e19e388de3464493a9e58d8ca26e8733; no voters:
I20260619 04:46:09.679601 11561 raft_consensus.cc:2781] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:46:09.679744 11561 raft_consensus.cc:482] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:46:09.679868 11561 raft_consensus.cc:3037] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:09.680004 11413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be192df15847c684cc5e5ad80a4b1c" candidate_uuid: "464f118ea75742279b82ad32617a1967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e19e388de3464493a9e58d8ca26e8733" is_pre_election: true
I20260619 04:46:09.682034 11561 raft_consensus.cc:504] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.682472 11413 raft_consensus.cc:2379] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 464f118ea75742279b82ad32617a1967 in current term 1: Already voted for candidate e19e388de3464493a9e58d8ca26e8733 in this term.
I20260619 04:46:09.682492 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.670393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:40913 (local address 127.3.243.129:38661)
0619 04:46:09.674907 (+ 4514us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.674927 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:09.675069 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:09.675696 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:09.675703 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:09.675792 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:09.676193 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.676206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.678310 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.678315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.680684 (+ 2369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:09.680695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.681758 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.681764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:09.681920 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.681947 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 04:46:09.682036 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 04:46:09.682154 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4382,"thread_start_us":4192,"threads_started":1}
I20260619 04:46:09.682528 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.670718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:38661 (local address 127.3.243.130:40913)
0619 04:46:09.674276 (+ 3558us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.674283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.675178 (+ 895us) server_negotiation.cc:407] Connection header received
0619 04:46:09.675230 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.675237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.675311 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.675464 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:09.676573 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.678137 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.680813 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.681556 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.681748 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.681819 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.682073 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.682195 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3387,"thread_start_us":3257,"threads_started":1}
I20260619 04:46:09.682960 11561 leader_election.cc:288] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [CANDIDATE]: Term 1 election: Requested vote from peers 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301), 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130:40913)
I20260619 04:46:09.683198 11272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be192df15847c684cc5e5ad80a4b1c" candidate_uuid: "464f118ea75742279b82ad32617a1967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91cb2bca3e284632bf8cd12eacfb2819" is_pre_election: true
I20260619 04:46:09.683331 11127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be192df15847c684cc5e5ad80a4b1c" candidate_uuid: "e19e388de3464493a9e58d8ca26e8733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464f118ea75742279b82ad32617a1967"
I20260619 04:46:09.683424 11272 raft_consensus.cc:2454] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 464f118ea75742279b82ad32617a1967 in term 0.
I20260619 04:46:09.683542 11127 raft_consensus.cc:3037] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:09.683825 11271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42be192df15847c684cc5e5ad80a4b1c" candidate_uuid: "e19e388de3464493a9e58d8ca26e8733" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91cb2bca3e284632bf8cd12eacfb2819"
I20260619 04:46:09.683892 11063 leader_election.cc:302] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 464f118ea75742279b82ad32617a1967, 91cb2bca3e284632bf8cd12eacfb2819; no voters: e19e388de3464493a9e58d8ca26e8733
I20260619 04:46:09.684099 11271 raft_consensus.cc:3037] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:09.686566 11127 raft_consensus.cc:2454] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e19e388de3464493a9e58d8ca26e8733 in term 1.
I20260619 04:46:09.686740 11558 raft_consensus.cc:2741] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 04:46:09.686955 11348 leader_election.cc:302] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 464f118ea75742279b82ad32617a1967, e19e388de3464493a9e58d8ca26e8733; no voters:
I20260619 04:46:09.687124 11271 raft_consensus.cc:2454] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e19e388de3464493a9e58d8ca26e8733 in term 1.
I20260619 04:46:09.687291 11561 raft_consensus.cc:2781] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:09.687850 11561 raft_consensus.cc:686] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 1 LEADER]: Becoming Leader. State: Replica: e19e388de3464493a9e58d8ca26e8733, State: Running, Role: LEADER
I20260619 04:46:09.688534 11561 consensus_queue.cc:237] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } }
I20260619 04:46:09.688830 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.688417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.131:50275)
0619 04:46:09.688603 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.688682 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 04:46:09.692045 10993 catalog_manager.cc:5495] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 reported cstate change: term changed from 0 to 1, leader changed from <none> to e19e388de3464493a9e58d8ca26e8733 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "e19e388de3464493a9e58d8ca26e8733" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e19e388de3464493a9e58d8ca26e8733" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 } health_report { overall_health: UNKNOWN } } }
I20260619 04:46:09.711221 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.706558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.3.243.131:44163)
0619 04:46:09.706712 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.706717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.706733 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:46:09.706875 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.706879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.706966 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.707085 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:46:09.707360 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.708160 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.709328 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.709750 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.709838 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.710190 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:46:09.710738 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:46:09.710840 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.710925 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.710987 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 04:46:09.715102 11127 raft_consensus.cc:1261] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 1 FOLLOWER]: Refusing update from remote peer e19e388de3464493a9e58d8ca26e8733: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:09.715423 11271 raft_consensus.cc:1261] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 1 FOLLOWER]: Refusing update from remote peer e19e388de3464493a9e58d8ca26e8733: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:09.715819 11570 consensus_queue.cc:1035] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464f118ea75742279b82ad32617a1967" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:09.716234 11561 consensus_queue.cc:1035] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91cb2bca3e284632bf8cd12eacfb2819" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 40913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:09.717568 11172 heartbeater.cc:503] Master 127.3.243.188:45313 was elected leader, sending a full tablet report...
I20260619 04:46:09.718842 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.718242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.129:45759)
0619 04:46:09.718525 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.718661 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 04:46:09.720045 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.719444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.130:51937)
0619 04:46:09.719671 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.719814 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 04:46:09.729122 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.728581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.131:60865)
0619 04:46:09.728844 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.728944 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 04:46:09.729965 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.729458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.129:34041)
0619 04:46:09.729665 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.729775 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 04:46:09.730654 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.730204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.3.243.130:55215)
0619 04:46:09.730404 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.730501 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 04:46:09.734771 11573 mvcc.cc:205] Tried to move back new op lower bound from 7298434538346643456 to 7298434538248073216. Current Snapshot: MvccSnapshot[applied={T|T < 7298434538346643456 or (T in {7298434538346643456})}]
I20260619 04:46:09.742830 10992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:09.743151 10992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:09.749701 10992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f80b1281aa2145729edb643968509225: Sending DeleteTablet for 3 replicas of tablet 42be192df15847c684cc5e5ad80a4b1c
I20260619 04:46:09.750530 11393 tablet_service.cc:1514] Processing DeleteTablet for tablet 42be192df15847c684cc5e5ad80a4b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:09 UTC) from {username='oryx', principal='oryx/127.3.243.188@KRBTEST.COM'} at 127.0.0.1:47888
I20260619 04:46:09.750669 11107 tablet_service.cc:1514] Processing DeleteTablet for tablet 42be192df15847c684cc5e5ad80a4b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:09 UTC) from {username='oryx', principal='oryx/127.3.243.188@KRBTEST.COM'} at 127.0.0.1:55928
I20260619 04:46:09.750674 11252 tablet_service.cc:1514] Processing DeleteTablet for tablet 42be192df15847c684cc5e5ad80a4b1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:09 UTC) from {username='oryx', principal='oryx/127.3.243.188@KRBTEST.COM'} at 127.0.0.1:51106
I20260619 04:46:09.754444 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 11010
I20260619 04:46:09.765993 11582 tablet_replica.cc:330] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: stopping tablet replica
I20260619 04:46:09.766201 11582 raft_consensus.cc:2229] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:46:09.766821 11582 raft_consensus.cc:2258] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:09.767814 11582 ts_tablet_manager.cc:1905] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:09.771025 11583 tablet_replica.cc:330] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: stopping tablet replica
I20260619 04:46:09.771278 11583 raft_consensus.cc:2229] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:09.771637 11583 raft_consensus.cc:2258] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:09.772360 11584 tablet_replica.cc:330] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: stopping tablet replica
I20260619 04:46:09.772588 11584 raft_consensus.cc:2229] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:09.772897 11584 raft_consensus.cc:2258] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:09.774150 11583 ts_tablet_manager.cc:1905] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:09.774822 11582 ts_tablet_manager.cc:1918] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:09.774950 11582 log.cc:1196] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/42be192df15847c684cc5e5ad80a4b1c
I20260619 04:46:09.775429 11582 ts_tablet_manager.cc:1939] T 42be192df15847c684cc5e5ad80a4b1c P e19e388de3464493a9e58d8ca26e8733: Deleting consensus metadata
I20260619 04:46:09.776685 10979 catalog_manager.cc:4841] TS e19e388de3464493a9e58d8ca26e8733 (127.3.243.131:44163): tablet 42be192df15847c684cc5e5ad80a4b1c (table test-table [id=b2805cbe9fa8484f9f2c7c483df8b468]) successfully deleted
I20260619 04:46:09.777040 11584 ts_tablet_manager.cc:1905] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:09.782801 11584 ts_tablet_manager.cc:1918] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:09.782961 11584 log.cc:1196] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/42be192df15847c684cc5e5ad80a4b1c
I20260619 04:46:09.783376 11584 ts_tablet_manager.cc:1939] T 42be192df15847c684cc5e5ad80a4b1c P 464f118ea75742279b82ad32617a1967: Deleting consensus metadata
I20260619 04:46:09.784621 10979 catalog_manager.cc:4841] TS 464f118ea75742279b82ad32617a1967 (127.3.243.129:34301): tablet 42be192df15847c684cc5e5ad80a4b1c (table test-table [id=b2805cbe9fa8484f9f2c7c483df8b468]) successfully deleted
I20260619 04:46:09.785063 11583 ts_tablet_manager.cc:1918] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:09.785187 11583 log.cc:1196] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/42be192df15847c684cc5e5ad80a4b1c
I20260619 04:46:09.785656 11583 ts_tablet_manager.cc:1939] T 42be192df15847c684cc5e5ad80a4b1c P 91cb2bca3e284632bf8cd12eacfb2819: Deleting consensus metadata
I20260619 04:46:09.786717 10980 catalog_manager.cc:4841] TS 91cb2bca3e284632bf8cd12eacfb2819 (127.3.243.130:40913): tablet 42be192df15847c684cc5e5ad80a4b1c (table test-table [id=b2805cbe9fa8484f9f2c7c483df8b468]) successfully deleted
I20260619 04:46:09.787678 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.756252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55936 (local address 127.3.243.129:34301)
0619 04:46:09.756470 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.756479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.760430 (+ 3951us) server_negotiation.cc:407] Connection header received
0619 04:46:09.760666 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.760675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.760784 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.760906 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:09.772470 (+ 11564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.773699 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.775529 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.776154 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.776287 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.779721 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:09.779739 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:09.779760 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:09.779808 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:09.784400 (+ 4592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:09.785347 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:09.785356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:09.785358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:09.785485 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:09.785896 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:09.785904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:09.785909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:09.786263 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:09.786914 (+ 651us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.787082 (+ 168us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.787287 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:46:09.788355 11167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:09.883198 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 11010
I20260619 04:46:09.902282 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 11184
I20260619 04:46:09.916821 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.903513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.3.243.130:40913)
0619 04:46:09.903708 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:09.903714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:09.903731 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:46:09.903791 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:09.903801 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:09.903859 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:09.903965 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:09.905479 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.906225 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.907513 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:09.907986 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:09.908165 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:09.910786 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:09.910819 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:09.910835 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:09.910870 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:09.913983 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:09.914679 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:09.914687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:09.914690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:09.914823 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:09.915181 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:09.915187 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:09.915190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:09.915591 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:09.915891 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:09.916239 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 04:46:09.916458 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:46:09.917492 11312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:09.962610 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.961707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60592)
0619 04:46:09.962263 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.962400 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":282,"threads_started":1}
I20260619 04:46:09.973438 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:09.972958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60606)
0619 04:46:09.973149 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:09.973265 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 04:46:10.047502 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 11184
I20260619 04:46:10.069582 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 11330
I20260619 04:46:10.085842 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:10.070427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47900 (local address 127.3.243.131:44163)
0619 04:46:10.070644 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:10.070650 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:10.070674 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:46:10.070802 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:10.070810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:10.070938 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:10.071058 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:10.072455 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:10.073648 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:10.074986 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:10.075746 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:10.075881 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:10.078413 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:10.078433 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:10.078455 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:10.078496 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:10.082881 (+ 4385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:10.083513 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:10.083522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:10.083526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:10.083689 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:10.084028 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:10.084033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:10.084045 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:10.084677 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:10.085031 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:10.085237 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 04:46:10.085467 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 04:46:10.086540 11453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:10.223798 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 11330
I20260619 04:46:10.244319 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 10666
I20260619 04:46:10.245420 10833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:10.360726 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:10.360055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60614)
0619 04:46:10.360414 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:10.360519 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260619 04:46:10.361776 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:10.361419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:34723 (local address 127.0.0.1:60622)
0619 04:46:10.361570 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:10.361660 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:34723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 04:46:10.364321 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 10666
I20260619 04:46:10.385176 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 10767
I20260619 04:46:10.386276 11024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:10.507167 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 10767
2026-06-19T04:46:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9645 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:46:10 dist-test-slave-2qp8 krb5kdc[11595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:46:10 dist-test-slave-2qp8 krb5kdc[11595](info): set up 2 sockets
Jun 19 04:46:10 dist-test-slave-2qp8 krb5kdc[11595](info): commencing operation
krb5kdc: starting...
W20260619 04:46:11.605228 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:46:11 dist-test-slave-2qp8 krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:46:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.190.keytab.
Entry for principal oryx/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.190.keytab.
I20260619 04:46:11.815376 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45455
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.190.keytab
--principal=oryx/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:45455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:12.146087 11608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:12.146472 11608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:12.146648 11608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:12.154214 11608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:12.154372 11608 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 04:46:12.154471 11608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:12.154590 11608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:12.154677 11608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:12.154772 11608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:12.165160 11608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:45455
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.190.keytab
--principal=oryx/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:12.166517 11608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:12.167923 11608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:12.180411 11613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:12.180327 11616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:12.180467 11614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:12.185765 11608 server_base.cc:1034] running on GCE node
I20260619 04:46:12.186719 11608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:12.188690 11608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:12.189994 11608 hybrid_clock.cc:648] HybridClock initialized: now 1781844372189899 us; error 61 us; skew 500 ppm
I20260619 04:46:12.190366 11608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:12 dist-test-slave-2qp8 krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:12.193753 11608 init.cc:377] Logged in from keytab as oryx/127.3.243.190@KRBTEST.COM (short username oryx)
I20260619 04:46:12.195345 11608 webserver.cc:460] Webserver started at http://127.3.243.190:46519/ using document root <none> and password file <none>
I20260619 04:46:12.196080 11608 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:12.196262 11608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:12.196619 11608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:12.199594 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "b93babe93aba4b2a9d4eb6af7a2e991a"
format_stamp: "Formatted at 2026-06-19 04:46:12 on dist-test-slave-2qp8"
server_key: "e5e75239efec17e30ed17e48457c5a4a"
server_key_iv: "be2176930b47fc44fc53d090ec036a1c"
server_key_version: "encryptionkey@0"
I20260619 04:46:12.200419 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "b93babe93aba4b2a9d4eb6af7a2e991a"
format_stamp: "Formatted at 2026-06-19 04:46:12 on dist-test-slave-2qp8"
server_key: "e5e75239efec17e30ed17e48457c5a4a"
server_key_iv: "be2176930b47fc44fc53d090ec036a1c"
server_key_version: "encryptionkey@0"
I20260619 04:46:12.205420 11608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:46:12.208906 11623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:12.210253 11608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:46:12.210577 11608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "b93babe93aba4b2a9d4eb6af7a2e991a"
format_stamp: "Formatted at 2026-06-19 04:46:12 on dist-test-slave-2qp8"
server_key: "e5e75239efec17e30ed17e48457c5a4a"
server_key_iv: "be2176930b47fc44fc53d090ec036a1c"
server_key_version: "encryptionkey@0"
I20260619 04:46:12.210803 11608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:12.224774 11608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:12.228781 11608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:12.229084 11608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:12.251238 11608 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:45455
I20260619 04:46:12.251251 11684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:45455 every 8 connection(s)
I20260619 04:46:12.252885 11608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:46:12.256466 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 11608
I20260619 04:46:12.256671 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:46:12.256844 11685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:12.257493 4046 external_mini_cluster.cc:1428] Setting key cfcd7813c5c63dc924fb54626f567060
I20260619 04:46:12.271402 11685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a: Bootstrap starting.
Jun 19 04:46:12 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:12.274852 11685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:12.276250 11685 log.cc:824] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:12.279752 11685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a: No bootstrap required, opened a new log
I20260619 04:46:12.281543 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:12.259515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.3.243.190:45455)
0619 04:46:12.260392 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:12.260402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:12.260445 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:46:12.262022 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:12.262044 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:12.262411 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:12.262729 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:12.264548 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.265742 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.267436 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.268186 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.268329 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:12.272995 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:12.273006 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:12.273019 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:12.273053 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:12.276805 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:12.277763 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:12.277771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:12.277779 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:12.277971 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:12.278376 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:12.278380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:12.278383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:12.279042 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:12.279327 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:12.279731 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 04:46:12.280187 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":447,"threads_started":1}
I20260619 04:46:12.288662 11685 raft_consensus.cc:348] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93babe93aba4b2a9d4eb6af7a2e991a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45455 } }
I20260619 04:46:12.289008 11685 raft_consensus.cc:374] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:12.289149 11685 raft_consensus.cc:729] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93babe93aba4b2a9d4eb6af7a2e991a, State: Initialized, Role: FOLLOWER
I20260619 04:46:12.289809 11685 consensus_queue.cc:260] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93babe93aba4b2a9d4eb6af7a2e991a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45455 } }
I20260619 04:46:12.290108 11685 raft_consensus.cc:388] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:46:12.290253 11685 raft_consensus.cc:482] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:46:12.290445 11685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:12.292768 11685 raft_consensus.cc:504] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93babe93aba4b2a9d4eb6af7a2e991a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45455 } }
I20260619 04:46:12.293336 11685 leader_election.cc:302] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b93babe93aba4b2a9d4eb6af7a2e991a; no voters:
I20260619 04:46:12.293869 11685 leader_election.cc:288] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:46:12.293977 11690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:12.294718 11690 raft_consensus.cc:686] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [term 1 LEADER]: Becoming Leader. State: Replica: b93babe93aba4b2a9d4eb6af7a2e991a, State: Running, Role: LEADER
I20260619 04:46:12.295284 11690 consensus_queue.cc:237] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93babe93aba4b2a9d4eb6af7a2e991a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45455 } }
I20260619 04:46:12.296044 11685 sys_catalog.cc:564] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:46:12.297952 11691 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b93babe93aba4b2a9d4eb6af7a2e991a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93babe93aba4b2a9d4eb6af7a2e991a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45455 } } }
I20260619 04:46:12.298409 11691 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:12.297943 11692 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b93babe93aba4b2a9d4eb6af7a2e991a. Latest consensus state: current_term: 1 leader_uuid: "b93babe93aba4b2a9d4eb6af7a2e991a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93babe93aba4b2a9d4eb6af7a2e991a" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45455 } } }
I20260619 04:46:12.300287 11692 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:12.302019 11696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:46:12.306715 11696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:46:12.314797 11696 catalog_manager.cc:1347] Generated new cluster ID: 73a4b08f8cb044b2bfb670ebce387ed5
I20260619 04:46:12.314926 11696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:46:12.332388 11696 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:46:12.333639 11696 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:46:12.346470 11696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a: Generated new TSK 0
I20260619 04:46:12.347338 11696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab.
Entry for principal oryx/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab.
I20260619 04:46:12.403460 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45455
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab
--principal=oryx/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:12.724223 11711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:12.724503 11711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:12.724671 11711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:12.731748 11711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:12.731871 11711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:12.732031 11711 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:46:12.732177 11711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:46:12.741689 11711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab
--principal=oryx/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.190:45455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:12.742846 11711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:12.744158 11711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:12.756775 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:12.756795 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:12.756930 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:12.763023 11711 server_base.cc:1034] running on GCE node
I20260619 04:46:12.763664 11711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:12.764611 11711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:12.765820 11711 hybrid_clock.cc:648] HybridClock initialized: now 1781844372765740 us; error 53 us; skew 500 ppm
I20260619 04:46:12.766120 11711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:12 dist-test-slave-2qp8 krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:12.769264 11711 init.cc:377] Logged in from keytab as oryx/127.3.243.129@KRBTEST.COM (short username oryx)
I20260619 04:46:12.770656 11711 webserver.cc:460] Webserver started at http://127.3.243.129:34971/ using document root <none> and password file <none>
I20260619 04:46:12.771355 11711 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:12.771466 11711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:12.771790 11711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:12.774713 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9"
format_stamp: "Formatted at 2026-06-19 04:46:12 on dist-test-slave-2qp8"
server_key: "cb2aa9374eabb7dc69e94d4757dfe689"
server_key_iv: "6c2f85e44a02f70cf6e210afafb2b9e9"
server_key_version: "encryptionkey@0"
I20260619 04:46:12.775509 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9"
format_stamp: "Formatted at 2026-06-19 04:46:12 on dist-test-slave-2qp8"
server_key: "cb2aa9374eabb7dc69e94d4757dfe689"
server_key_iv: "6c2f85e44a02f70cf6e210afafb2b9e9"
server_key_version: "encryptionkey@0"
I20260619 04:46:12.780527 11711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 04:46:12.783725 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:12.785028 11711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:46:12.785197 11711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9"
format_stamp: "Formatted at 2026-06-19 04:46:12 on dist-test-slave-2qp8"
server_key: "cb2aa9374eabb7dc69e94d4757dfe689"
server_key_iv: "6c2f85e44a02f70cf6e210afafb2b9e9"
server_key_version: "encryptionkey@0"
I20260619 04:46:12.785419 11711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:12.815055 11711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:12.819021 11711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:12.819307 11711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:12.821924 11711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:12.822041 11711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:12.822177 11711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:12.822244 11711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:46:12 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:12.846699 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:12.822782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:53993 (local address 127.3.243.190:45455)
0619 04:46:12.823293 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:12.823301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:12.824559 (+ 1258us) server_negotiation.cc:407] Connection header received
0619 04:46:12.826607 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:12.826616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:12.826699 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:12.826828 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:12.829098 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.830161 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.832955 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.833514 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.833749 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:12.839176 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:12.839186 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:12.839193 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:12.839262 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:12.842895 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:12.843925 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:12.843930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:12.843935 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:12.844113 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:12.844789 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:12.844795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:12.844799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:12.845172 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:12.845331 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:12.846079 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 04:46:12.846316 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":282,"threads_started":1}
I20260619 04:46:12.847731 11736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:12.823521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45455 (local address 127.3.243.129:53993)
0619 04:46:12.824398 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:12.824445 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:12.826285 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:12.827080 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:12.827094 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:12.827921 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:12.828918 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:12.828941 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.830395 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.830400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:12.832787 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:12.832814 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.833952 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.833962 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:12.834205 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:12.835022 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:12.835049 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:12.838895 (+ 3846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:12.843089 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:12.843102 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:12.843124 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:12.843799 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:12.844255 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:12.844259 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:12.844263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:12.844634 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:12.845332 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:12.845341 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:12.845702 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 04:46:12.845963 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 04:46:12.846505 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":351,"threads_started":1}
I20260619 04:46:12.868635 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:12.854345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:44623 (local address 127.3.243.190:45455)
0619 04:46:12.854602 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:12.854608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:12.854801 (+ 193us) server_negotiation.cc:407] Connection header received
0619 04:46:12.854962 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:12.854970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:12.855069 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:12.855191 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:12.855970 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.857158 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.858436 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.859175 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.859322 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:12.861191 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:12.861201 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:12.861209 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:12.861264 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:12.865494 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:12.866161 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:12.866169 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:12.866176 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:12.866405 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:12.866917 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:12.866924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:12.866931 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:12.867392 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:12.867567 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:12.868034 (+ 467us) server_negotiation.cc:299] Negotiation successful
0619 04:46:12.868276 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260619 04:46:12.868782 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:12.853945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45455 (local address 127.3.243.129:44623)
0619 04:46:12.854695 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:12.854714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:12.854845 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:12.855197 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:12.855200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:12.855577 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:12.855834 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:12.855842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.857285 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.857288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:12.858305 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:12.858316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:12.859369 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:12.859376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:12.859571 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:12.860246 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:12.860262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:12.860939 (+ 677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:12.865655 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:12.865658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:12.865661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:12.866043 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:12.866563 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:12.866569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:12.866574 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:12.866781 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:12.867527 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:12.867533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:12.867768 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 04:46:12.867945 (+ 177us) client_negotiation.cc:240] Negotiation successful
0619 04:46:12.868298 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":648,"threads_started":1}
I20260619 04:46:12.877851 11711 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:40961
I20260619 04:46:12.877866 11846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:40961 every 8 connection(s)
I20260619 04:46:12.879168 11711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:46:12.881392 11847 heartbeater.cc:347] Connected to a master server at 127.3.243.190:45455
I20260619 04:46:12.881623 11847 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:12.882273 11847 heartbeater.cc:511] Master 127.3.243.190:45455 requested a full tablet report, sending...
I20260619 04:46:12.884280 11640 ts_manager.cc:194] Registered new tserver with Master: ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961)
I20260619 04:46:12.885253 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 11711
I20260619 04:46:12.885489 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:46:12.885838 11640 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:53993
I20260619 04:46:12.886147 4046 external_mini_cluster.cc:1428] Setting key e100831d64819df643c3676d7df5cca3
WARNING: no policy specified for oryx/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab.
Entry for principal oryx/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab.
I20260619 04:46:12.931583 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45455
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab
--principal=oryx/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:13.249991 11851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:13.250269 11851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:13.250427 11851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:13.261924 11851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:13.262059 11851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:13.262187 11851 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:46:13.262315 11851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:46:13.276423 11851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab
--principal=oryx/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.190:45455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:13.277583 11851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:13.278816 11851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:13.291252 11856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:13.291292 11857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:13.291334 11859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:13.292443 11851 server_base.cc:1034] running on GCE node
I20260619 04:46:13.293030 11851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:13.293874 11851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:13.295115 11851 hybrid_clock.cc:648] HybridClock initialized: now 1781844373295053 us; error 38 us; skew 500 ppm
I20260619 04:46:13.295406 11851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:13 dist-test-slave-2qp8 krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:13.298457 11851 init.cc:377] Logged in from keytab as oryx/127.3.243.130@KRBTEST.COM (short username oryx)
I20260619 04:46:13.299777 11851 webserver.cc:460] Webserver started at http://127.3.243.130:43303/ using document root <none> and password file <none>
I20260619 04:46:13.300554 11851 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:13.300671 11851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:13.300976 11851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:13.303877 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "dda69be6a15746f9819322d73dec9a81"
format_stamp: "Formatted at 2026-06-19 04:46:13 on dist-test-slave-2qp8"
server_key: "0d0393cf3b2ee380fba4ade70ad6f748"
server_key_iv: "d329a557cdcf8202728a22b008a5d00f"
server_key_version: "encryptionkey@0"
I20260619 04:46:13.304719 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "dda69be6a15746f9819322d73dec9a81"
format_stamp: "Formatted at 2026-06-19 04:46:13 on dist-test-slave-2qp8"
server_key: "0d0393cf3b2ee380fba4ade70ad6f748"
server_key_iv: "d329a557cdcf8202728a22b008a5d00f"
server_key_version: "encryptionkey@0"
I20260619 04:46:13.309614 11851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 04:46:13.312969 11866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:13.314247 11851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:46:13.314409 11851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "dda69be6a15746f9819322d73dec9a81"
format_stamp: "Formatted at 2026-06-19 04:46:13 on dist-test-slave-2qp8"
server_key: "0d0393cf3b2ee380fba4ade70ad6f748"
server_key_iv: "d329a557cdcf8202728a22b008a5d00f"
server_key_version: "encryptionkey@0"
I20260619 04:46:13.314631 11851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:13.336082 11851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:13.340035 11851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:13.340330 11851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:13.342970 11851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:13.343096 11851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:13.343233 11851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:13.343298 11851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:46:13 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:13.367734 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:13.343897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:36273 (local address 127.3.243.190:45455)
0619 04:46:13.344083 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:13.344088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:13.345517 (+ 1429us) server_negotiation.cc:407] Connection header received
0619 04:46:13.347513 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:13.347519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:13.347605 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:13.347735 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:13.350357 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.351404 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.354593 (+ 3189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.355179 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.355323 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:13.359890 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:13.359900 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:13.359908 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:13.359980 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:13.364239 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:13.365229 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:13.365235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:13.365239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:13.365362 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:13.365957 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:13.365962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:13.365966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:13.366260 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:13.366416 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:13.367144 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 04:46:13.367370 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 04:46:13.369344 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:13.344554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45455 (local address 127.3.243.130:36273)
0619 04:46:13.345353 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:13.345393 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:13.347232 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:13.348037 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:13.348057 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:13.349001 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:13.350145 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:13.350176 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.351566 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.351575 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:13.354420 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:13.354449 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.355470 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.355490 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:13.355849 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:13.356640 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:13.356661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:13.359597 (+ 2936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:13.364420 (+ 4823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:13.364432 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:13.364450 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:13.365090 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:13.365524 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:13.365531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:13.365537 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:13.365832 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:13.366390 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:13.366398 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:13.366769 (+ 371us) client_negotiation.cc:769] Sending connection context
0619 04:46:13.367042 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 04:46:13.367609 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":331,"threads_started":1}
I20260619 04:46:13.392545 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:13.376096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:42241 (local address 127.3.243.190:45455)
0619 04:46:13.376332 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:13.376340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:13.377179 (+ 839us) server_negotiation.cc:407] Connection header received
0619 04:46:13.377386 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:13.377395 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:13.377483 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:13.377594 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:13.379425 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.380447 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.383018 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.383422 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.383524 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:13.386260 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:13.386269 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:13.386276 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:13.386318 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:13.389315 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:13.390323 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:13.390326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:13.390328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:13.390449 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:13.390881 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:13.390884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:13.390886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:13.391185 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:13.391290 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:13.391846 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 04:46:13.392093 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:46:13.392897 11930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:13.376308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45455 (local address 127.3.243.130:42241)
0619 04:46:13.377078 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:13.377094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:13.377260 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:13.377654 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:13.377664 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:13.378660 (+ 996us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:13.379256 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:13.379277 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.380568 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.380576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:13.382856 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:13.382876 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.383628 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.383637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:13.383906 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:13.384720 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:13.384740 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:13.385999 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:13.389446 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:13.389451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:13.389456 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:13.390201 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:13.390562 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:13.390570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:13.390575 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:13.390767 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:13.391294 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:13.391301 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:13.391544 (+ 243us) client_negotiation.cc:769] Sending connection context
0619 04:46:13.391731 (+ 187us) client_negotiation.cc:240] Negotiation successful
0619 04:46:13.392214 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":482,"threads_started":1}
I20260619 04:46:13.401448 11851 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:35985
I20260619 04:46:13.401468 11985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:35985 every 8 connection(s)
I20260619 04:46:13.402735 11851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:46:13.404831 11986 heartbeater.cc:347] Connected to a master server at 127.3.243.190:45455
I20260619 04:46:13.405071 11986 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:13.405705 11986 heartbeater.cc:511] Master 127.3.243.190:45455 requested a full tablet report, sending...
I20260619 04:46:13.407029 11640 ts_manager.cc:194] Registered new tserver with Master: dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985)
I20260619 04:46:13.407841 11640 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:36273
I20260619 04:46:13.412456 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 11851
I20260619 04:46:13.412590 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:46:13.412994 4046 external_mini_cluster.cc:1428] Setting key 2729b9e51104c9aad18e87cd20fcdd62
WARNING: no policy specified for oryx/127.3.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab.
Entry for principal oryx/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab.
I20260619 04:46:13.459822 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45455
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab
--principal=oryx/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:13.776355 11990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:13.776669 11990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:13.776882 11990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:13.784754 11990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:13.784866 11990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:13.785063 11990 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 04:46:13.785194 11990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:46:13.796327 11990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab
--principal=oryx/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.243.190:45455
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:13.797890 11990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:13.799371 11990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:13.813796 11995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:13.813838 11996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:13.813882 11998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:13.814851 11990 server_base.cc:1034] running on GCE node
I20260619 04:46:13.815423 11990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:13.816455 11990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:13.817688 11990 hybrid_clock.cc:648] HybridClock initialized: now 1781844373817629 us; error 39 us; skew 500 ppm
I20260619 04:46:13.818006 11990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:13 dist-test-slave-2qp8 krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:13.821120 11990 init.cc:377] Logged in from keytab as oryx/127.3.243.131@KRBTEST.COM (short username oryx)
I20260619 04:46:13.822487 11990 webserver.cc:460] Webserver started at http://127.3.243.131:34251/ using document root <none> and password file <none>
I20260619 04:46:13.823181 11990 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:13.823308 11990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:13.823630 11990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:13.826521 11990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9"
format_stamp: "Formatted at 2026-06-19 04:46:13 on dist-test-slave-2qp8"
server_key: "ca4d51636dc64fd1bad267a36f49f6a0"
server_key_iv: "2505a4efec8c80813eebd4170710b421"
server_key_version: "encryptionkey@0"
I20260619 04:46:13.827356 11990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9"
format_stamp: "Formatted at 2026-06-19 04:46:13 on dist-test-slave-2qp8"
server_key: "ca4d51636dc64fd1bad267a36f49f6a0"
server_key_iv: "2505a4efec8c80813eebd4170710b421"
server_key_version: "encryptionkey@0"
I20260619 04:46:13.832998 11990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 04:46:13.836377 12005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:13.837673 11990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:46:13.837848 11990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9"
format_stamp: "Formatted at 2026-06-19 04:46:13 on dist-test-slave-2qp8"
server_key: "ca4d51636dc64fd1bad267a36f49f6a0"
server_key_iv: "2505a4efec8c80813eebd4170710b421"
server_key_version: "encryptionkey@0"
I20260619 04:46:13.838078 11990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:13.857458 11990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:13.861763 11990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:13.862178 11990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:13.865208 11990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:13.865324 11990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:13.865509 11990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:13.865622 11990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Jun 19 04:46:13 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844373, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:13.889036 11847 heartbeater.cc:503] Master 127.3.243.190:45455 was elected leader, sending a full tablet report...
I20260619 04:46:13.892851 12014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:13.866439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45455 (local address 127.3.243.131:58737)
0619 04:46:13.867309 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:13.867375 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:13.869411 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:13.870622 (+ 1211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:13.870638 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:13.871591 (+ 953us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:13.872775 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:13.872813 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.874328 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.874333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:13.876594 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:13.876609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.877765 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.877772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:13.878055 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:13.878986 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:13.879021 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:13.882302 (+ 3281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:13.887224 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:13.887253 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:13.887298 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:13.888268 (+ 970us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:13.888792 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:13.888797 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:13.888800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:13.889015 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:13.889764 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:13.889772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:13.890353 (+ 581us) client_negotiation.cc:769] Sending connection context
0619 04:46:13.890685 (+ 332us) client_negotiation.cc:240] Negotiation successful
0619 04:46:13.891185 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":384,"threads_started":1}
I20260619 04:46:13.893862 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:13.865777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:58737 (local address 127.3.243.190:45455)
0619 04:46:13.866025 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:13.866035 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:13.867513 (+ 1478us) server_negotiation.cc:407] Connection header received
0619 04:46:13.869776 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:13.869782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:13.869883 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:13.870011 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:13.873057 (+ 3046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.874154 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.876757 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.877319 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.877456 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:13.882635 (+ 5179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:13.882645 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:13.882653 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:13.882710 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:13.887033 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:13.888480 (+ 1447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:13.888488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:13.888492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:13.888652 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:13.889151 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:13.889158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:13.889166 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:13.889493 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:13.889683 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:13.893457 (+ 3774us) server_negotiation.cc:299] Negotiation successful
0619 04:46:13.893632 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:46:13.925575 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:13.907753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:47223 (local address 127.3.243.190:45455)
0619 04:46:13.907946 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:13.907954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:13.909256 (+ 1302us) server_negotiation.cc:407] Connection header received
0619 04:46:13.909476 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:13.909501 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:13.909607 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:13.909711 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:13.911469 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.912698 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.914888 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.915601 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.915733 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:13.918554 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:13.918564 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:13.918573 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:13.918621 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:13.922664 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:13.923476 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:13.923489 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:13.923491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:13.923602 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:13.924117 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:13.924123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:13.924126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:13.924416 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:13.924582 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:13.924970 (+ 388us) server_negotiation.cc:299] Negotiation successful
0619 04:46:13.925195 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:46:13.925701 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:13.908063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45455 (local address 127.3.243.131:47223)
0619 04:46:13.909128 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:13.909176 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:13.909325 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:13.909914 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:13.909920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:13.910727 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:13.911253 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:13.911275 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.912892 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.912898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:13.914750 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:13.914761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.915959 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.915966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:13.916232 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:13.916973 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:13.916996 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:13.918285 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:13.922832 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:13.922843 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:13.922849 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:13.923348 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:13.923765 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:13.923772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:13.923777 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:13.923937 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:13.924550 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:13.924555 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:13.924734 (+ 179us) client_negotiation.cc:769] Sending connection context
0619 04:46:13.924876 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 04:46:13.925217 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":584,"threads_started":1}
I20260619 04:46:13.926488 11990 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:43235
I20260619 04:46:13.926542 12124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:43235 every 8 connection(s)
I20260619 04:46:13.928097 11990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:46:13.930464 12125 heartbeater.cc:347] Connected to a master server at 127.3.243.190:45455
I20260619 04:46:13.930689 12125 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:13.930702 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 11990
I20260619 04:46:13.930922 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:46:13.931406 12125 heartbeater.cc:511] Master 127.3.243.190:45455 requested a full tablet report, sending...
I20260619 04:46:13.931432 4046 external_mini_cluster.cc:1428] Setting key e0677b4947ec65fb90f84d894563dc8a
I20260619 04:46:13.932821 11640 ts_manager.cc:194] Registered new tserver with Master: cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235)
I20260619 04:46:13.933727 11640 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:58737
I20260619 04:46:13.934645 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 04:46:13 dist-test-slave-2qp8 krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 04:46:13 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:13 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:14.410245 11986 heartbeater.cc:503] Master 127.3.243.190:45455 was elected leader, sending a full tablet report...
I20260619 04:46:14.936219 12125 heartbeater.cc:503] Master 127.3.243.190:45455 was elected leader, sending a full tablet report...
W20260619 04:46:15.031006 12133 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.052s user 0.001s sys 0.014s
W20260619 04:46:15.032482 11735 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:13.973438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.3.243.190:45455)
0619 04:46:13.973667 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:13.973674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:13.973910 (+ 236us) server_negotiation.cc:407] Connection header received
0619 04:46:13.974126 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:13.974131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:13.974195 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:13.974301 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:13.975600 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.976399 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.977628 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:13.978057 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:13.978205 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.032048 (+1053843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55262: BlockingRecv error: recv got EOF from 127.0.0.1:55262 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 04:46:15 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844373, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:15.057494 11735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.036825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55270 (local address 127.3.243.190:45455)
0619 04:46:15.036990 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.036994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.037425 (+ 431us) server_negotiation.cc:407] Connection header received
0619 04:46:15.037557 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.037561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.037642 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.037732 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:15.038951 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.039682 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.040878 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.041333 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.041441 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.052297 (+ 10856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:15.052302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:15.052306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:15.052345 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:15.054732 (+ 2387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:15.055394 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:15.055401 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:15.055405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:15.055540 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:15.055900 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:15.055906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:15.055909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:15.056247 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:15.056578 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.056900 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.057149 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:46:15.061625 11639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:46:15.068615 11639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:46:15.106043 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.080382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38720 (local address 127.3.243.130:35985)
0619 04:46:15.097788 (+ 17406us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.097795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.097819 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:46:15.097914 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.097921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.098157 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.098375 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:15.099590 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.100963 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.103848 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.104896 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.105052 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.105273 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.105548 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.105708 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17269,"thread_start_us":17143,"threads_started":1}
I20260619 04:46:15.106127 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.080118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:35985 (local address 127.0.0.1:38720)
0619 04:46:15.090849 (+ 10731us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.090887 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.091076 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.098586 (+ 7510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.098595 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.098625 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:15.099395 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.099409 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.101101 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.101109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.103685 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.103701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.105090 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.105096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.105281 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.105345 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.105496 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.105703 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10426,"mutex_wait_us":119,"thread_start_us":16426,"threads_started":1}
I20260619 04:46:15.106796 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.080901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:40961 (local address 127.0.0.1:36354)
0619 04:46:15.096774 (+ 15873us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.096797 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.096934 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.098165 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.098170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.098221 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:15.098939 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.098948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.100723 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.100729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.103733 (+ 3004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.103748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.105729 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.105735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.105981 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.106142 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.106251 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.106387 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15493,"mutex_wait_us":297,"spinlock_wait_cycles":1408,"thread_start_us":9539,"threads_started":1}
I20260619 04:46:15.107060 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.080301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:43235 (local address 127.0.0.1:50698)
0619 04:46:15.092656 (+ 12355us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.092680 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.092825 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.098155 (+ 5330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.098164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.098209 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:15.098931 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.098945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.100569 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.100578 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.103739 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.103748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.105715 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.105722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.105984 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.106142 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.106251 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.106612 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12199,"mutex_wait_us":542,"spinlock_wait_cycles":4864,"thread_start_us":12062,"threads_started":1}
I20260619 04:46:15.107774 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.081569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.3.243.129:40961)
0619 04:46:15.097458 (+ 15889us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.097472 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.097508 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:46:15.097620 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.097629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.097907 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.098819 (+ 912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:15.099134 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.100585 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.103928 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.105441 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.107006 (+ 1565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.107232 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.107300 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.107460 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15657,"thread_start_us":16191,"threads_started":1}
I20260619 04:46:15.108075 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.080969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.3.243.131:43235)
0619 04:46:15.097456 (+ 16487us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.097469 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.097504 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:46:15.097663 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.097673 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.097915 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.099127 (+ 1212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:15.099333 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.100439 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.103928 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.105451 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.107342 (+ 1891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.107539 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.107616 (+ 77us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.107780 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16316,"thread_start_us":16193,"threads_started":1}
I20260619 04:46:15.110414 11916 tablet_service.cc:1467] Processing CreateTablet for tablet 07c618720751425eb07d4668167125ec (DEFAULT_TABLE table=test-table [id=9100cb40bde94e778360dcce5504370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:15.110841 12055 tablet_service.cc:1467] Processing CreateTablet for tablet 07c618720751425eb07d4668167125ec (DEFAULT_TABLE table=test-table [id=9100cb40bde94e778360dcce5504370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:15.111284 11777 tablet_service.cc:1467] Processing CreateTablet for tablet 07c618720751425eb07d4668167125ec (DEFAULT_TABLE table=test-table [id=9100cb40bde94e778360dcce5504370d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:15.111923 11916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c618720751425eb07d4668167125ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:15.112174 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c618720751425eb07d4668167125ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:15.112564 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07c618720751425eb07d4668167125ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:15.123732 12150 tablet_bootstrap.cc:492] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: Bootstrap starting.
I20260619 04:46:15.126760 12150 tablet_bootstrap.cc:654] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:15.127760 12150 log.cc:824] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:15.128934 12152 tablet_bootstrap.cc:492] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: Bootstrap starting.
I20260619 04:46:15.131827 12151 tablet_bootstrap.cc:492] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: Bootstrap starting.
I20260619 04:46:15.133287 12152 tablet_bootstrap.cc:654] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:15.133711 12150 tablet_bootstrap.cc:492] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: No bootstrap required, opened a new log
I20260619 04:46:15.134037 12150 ts_tablet_manager.cc:1397] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260619 04:46:15.134574 12152 log.cc:824] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:15.135502 12151 tablet_bootstrap.cc:654] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:15.136734 12151 log.cc:824] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:15.137673 12152 tablet_bootstrap.cc:492] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: No bootstrap required, opened a new log
I20260619 04:46:15.137952 12152 ts_tablet_manager.cc:1397] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:46:15.139650 12151 tablet_bootstrap.cc:492] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: No bootstrap required, opened a new log
I20260619 04:46:15.139950 12151 ts_tablet_manager.cc:1397] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260619 04:46:15.143553 12150 raft_consensus.cc:348] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.143922 12150 raft_consensus.cc:374] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:15.144024 12150 raft_consensus.cc:729] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0067a8d8d54d2b9b8d5ce4654c7da9, State: Initialized, Role: FOLLOWER
I20260619 04:46:15.144672 12150 consensus_queue.cc:260] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.146399 12150 ts_tablet_manager.cc:1428] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260619 04:46:15.148842 12152 raft_consensus.cc:348] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.149327 12152 raft_consensus.cc:374] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:15.149433 12152 raft_consensus.cc:729] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4ae7acc2eb4e97b3fcb8224e8281e9, State: Initialized, Role: FOLLOWER
I20260619 04:46:15.150146 12152 consensus_queue.cc:260] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.150890 12151 raft_consensus.cc:348] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.151363 12151 raft_consensus.cc:374] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:15.151486 12151 raft_consensus.cc:729] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda69be6a15746f9819322d73dec9a81, State: Initialized, Role: FOLLOWER
I20260619 04:46:15.152076 12151 consensus_queue.cc:260] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.152890 12152 ts_tablet_manager.cc:1428] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260619 04:46:15.154155 12151 ts_tablet_manager.cc:1428] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260619 04:46:15.155596 11987 tablet.cc:2369] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:46:15.180810 12126 tablet.cc:2369] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:46:15.329996 12158 raft_consensus.cc:482] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:15.330328 12158 raft_consensus.cc:504] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.330638 12157 raft_consensus.cc:482] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:15.330940 12157 raft_consensus.cc:504] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.332389 12158 leader_election.cc:288] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235), ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961)
I20260619 04:46:15.339708 12157 leader_election.cc:288] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985), cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235)
I20260619 04:46:15.343569 12161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.333025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:43235 (local address 127.3.243.129:40979)
0619 04:46:15.333743 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.333769 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.333947 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.335083 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.335091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.335133 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:15.335600 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.335616 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.337916 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.337924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.340639 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.340652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.342666 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.342672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.342860 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.342895 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.342991 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.343125 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":7159,"threads_started":1}
I20260619 04:46:15.344422 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.333495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:40979 (local address 127.3.243.131:43235)
0619 04:46:15.334609 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.334617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.334646 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:46:15.334733 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.334741 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.334839 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.335772 (+ 933us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:15.336148 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.337729 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.340841 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.342255 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.342437 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.342511 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.343852 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.344006 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":726,"threads_started":1}
I20260619 04:46:15.345506 12080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c618720751425eb07d4668167125ec" candidate_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" is_pre_election: true
I20260619 04:46:15.346104 12080 raft_consensus.cc:2454] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in term 0.
I20260619 04:46:15.346958 11728 leader_election.cc:302] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, ed4ae7acc2eb4e97b3fcb8224e8281e9; no voters:
I20260619 04:46:15.347510 12157 raft_consensus.cc:2781] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:46:15.347739 12157 raft_consensus.cc:482] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:46:15.347913 12157 raft_consensus.cc:3037] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:15.348443 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.339728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:57331 (local address 127.3.243.130:35985)
0619 04:46:15.339884 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.339889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.340579 (+ 690us) server_negotiation.cc:407] Connection header received
0619 04:46:15.340854 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.340859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.340952 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.341033 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:15.341884 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.343050 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.345873 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.347204 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.347349 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.347424 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.348016 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.348172 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":46}
I20260619 04:46:15.348489 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.332588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:35009 (local address 127.3.243.131:43235)
0619 04:46:15.332791 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.332799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.339887 (+ 7088us) server_negotiation.cc:407] Connection header received
0619 04:46:15.340998 (+ 1111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.341003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.341119 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.341183 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:15.342016 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.343182 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.346181 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.347407 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.347550 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.347614 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.348142 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.348271 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:46:15.349251 12159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.332596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:43235 (local address 127.3.243.130:35009)
0619 04:46:15.339778 (+ 7182us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.339802 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.340535 (+ 733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.341387 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.341391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.341413 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:15.341832 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.341844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.343391 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.343396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.345988 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.346002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.347743 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.347749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.347929 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.347960 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.348690 (+ 730us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.348840 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7036,"mutex_wait_us":173,"thread_start_us":6851,"threads_started":1}
I20260619 04:46:15.349467 12160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.333118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:35985 (local address 127.3.243.129:57331)
0619 04:46:15.340475 (+ 7357us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.340502 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.340690 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.341215 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.341219 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.341264 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:15.341707 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.341718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.343212 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.343217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.345662 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.345680 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.347605 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.347612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.347807 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.347846 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.348865 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.349009 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7213,"spinlock_wait_cycles":28544,"thread_start_us":278,"threads_started":1}
I20260619 04:46:15.349684 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.332491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:49401 (local address 127.3.243.129:40961)
0619 04:46:15.332824 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.332832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.341001 (+ 8169us) server_negotiation.cc:407] Connection header received
0619 04:46:15.341282 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.341287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.341464 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.341545 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:15.342521 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.343673 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.347103 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.348424 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.348563 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.348638 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.349199 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.349389 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":187,"spinlock_wait_cycles":24960}
I20260619 04:46:15.350039 12080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c618720751425eb07d4668167125ec" candidate_uuid: "dda69be6a15746f9819322d73dec9a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" is_pre_election: true
I20260619 04:46:15.350291 12080 raft_consensus.cc:2454] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dda69be6a15746f9819322d73dec9a81 in term 0.
I20260619 04:46:15.350508 12163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.333754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:40961 (local address 127.3.243.130:49401)
0619 04:46:15.340901 (+ 7147us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.340918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.341127 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.341818 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.341823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.341846 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:15.342322 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.342333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.343871 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.343879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.346922 (+ 3043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.346934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.348829 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.348834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.349007 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.349040 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.349744 (+ 704us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.350078 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7009,"mutex_wait_us":221,"spinlock_wait_cycles":4480,"thread_start_us":6869,"threads_started":1}
I20260619 04:46:15.350744 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c618720751425eb07d4668167125ec" candidate_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81" is_pre_election: true
I20260619 04:46:15.350936 11868 leader_election.cc:302] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, dda69be6a15746f9819322d73dec9a81; no voters:
I20260619 04:46:15.351178 11941 raft_consensus.cc:2454] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in term 0.
I20260619 04:46:15.351349 12158 raft_consensus.cc:2781] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:46:15.351490 12158 raft_consensus.cc:482] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:46:15.351588 12158 raft_consensus.cc:3037] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:15.351444 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c618720751425eb07d4668167125ec" candidate_uuid: "dda69be6a15746f9819322d73dec9a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" is_pre_election: true
I20260619 04:46:15.352872 12157 raft_consensus.cc:504] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.353329 11802 raft_consensus.cc:2379] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dda69be6a15746f9819322d73dec9a81 in current term 1: Already voted for candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in this term.
I20260619 04:46:15.353662 12157 leader_election.cc:288] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [CANDIDATE]: Term 1 election: Requested vote from peers dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985), cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235)
I20260619 04:46:15.354058 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c618720751425eb07d4668167125ec" candidate_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81"
I20260619 04:46:15.354259 12080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c618720751425eb07d4668167125ec" candidate_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9"
I20260619 04:46:15.354458 12080 raft_consensus.cc:3037] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:15.354702 12158 raft_consensus.cc:504] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.355160 11941 raft_consensus.cc:2379] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in current term 1: Already voted for candidate dda69be6a15746f9819322d73dec9a81 in this term.
I20260619 04:46:15.355443 12158 leader_election.cc:288] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [CANDIDATE]: Term 1 election: Requested vote from peers cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235), ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961)
I20260619 04:46:15.355870 12079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c618720751425eb07d4668167125ec" candidate_uuid: "dda69be6a15746f9819322d73dec9a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9"
I20260619 04:46:15.356017 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07c618720751425eb07d4668167125ec" candidate_uuid: "dda69be6a15746f9819322d73dec9a81" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9"
I20260619 04:46:15.356272 11802 raft_consensus.cc:2379] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dda69be6a15746f9819322d73dec9a81 in current term 1: Already voted for candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in this term.
I20260619 04:46:15.357009 11868 leader_election.cc:302] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dda69be6a15746f9819322d73dec9a81; no voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, ed4ae7acc2eb4e97b3fcb8224e8281e9
I20260619 04:46:15.357331 12158 raft_consensus.cc:2726] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 04:46:15.357607 12080 raft_consensus.cc:2454] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in term 1.
I20260619 04:46:15.357952 11728 leader_election.cc:302] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, ed4ae7acc2eb4e97b3fcb8224e8281e9; no voters: dda69be6a15746f9819322d73dec9a81
I20260619 04:46:15.358232 12157 raft_consensus.cc:2781] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:15.358700 12157 raft_consensus.cc:686] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 LEADER]: Becoming Leader. State: Replica: ed4ae7acc2eb4e97b3fcb8224e8281e9, State: Running, Role: LEADER
I20260619 04:46:15.359136 12157 consensus_queue.cc:237] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } }
I20260619 04:46:15.363332 11639 catalog_manager.cc:5495] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129). New cstate: current_term: 1 leader_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } health_report { overall_health: HEALTHY } } }
W20260619 04:46:15.383045 11848 tablet.cc:2369] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:46:15.395596 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.390343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.3.243.129:40961)
0619 04:46:15.390557 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.390563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.390669 (+ 106us) server_negotiation.cc:407] Connection header received
0619 04:46:15.390925 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.390928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.391002 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.391150 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:46:15.391523 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.392288 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.393572 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.393970 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.394099 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.394529 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:46:15.395016 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:46:15.395177 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.395227 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.395315 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 04:46:15.403621 12080 raft_consensus.cc:1261] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Refusing update from remote peer ed4ae7acc2eb4e97b3fcb8224e8281e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:15.403815 11941 raft_consensus.cc:1261] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Refusing update from remote peer ed4ae7acc2eb4e97b3fcb8224e8281e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:15.404498 12164 consensus_queue.cc:1035] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:15.404953 12157 consensus_queue.cc:1035] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:15.417145 12167 mvcc.cc:205] Tried to move back new op lower bound from 7298434561643003904 to 7298434561474912256. Current Snapshot: MvccSnapshot[applied={T|T < 7298434561643003904}]
I20260619 04:46:15.424620 12168 mvcc.cc:205] Tried to move back new op lower bound from 7298434561643003904 to 7298434561474912256. Current Snapshot: MvccSnapshot[applied={T|T < 7298434561643003904}]
I20260619 04:46:15.424913 12169 mvcc.cc:205] Tried to move back new op lower bound from 7298434561643003904 to 7298434561474912256. Current Snapshot: MvccSnapshot[applied={T|T < 7298434561643003904}]
I20260619 04:46:15.430882 11640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:15.431149 11640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:15.434861 11640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a: Sending DeleteTablet for 3 replicas of tablet 07c618720751425eb07d4668167125ec
I20260619 04:46:15.436028 11916 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c618720751425eb07d4668167125ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:15 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:38720
I20260619 04:46:15.436028 12055 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c618720751425eb07d4668167125ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:15 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:50698
I20260619 04:46:15.436257 11777 tablet_service.cc:1514] Processing DeleteTablet for tablet 07c618720751425eb07d4668167125ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:15 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:36354
I20260619 04:46:15.437714 11640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:46:15.438660 11640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:46:15.440685 12176 tablet_replica.cc:330] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: stopping tablet replica
I20260619 04:46:15.441249 12176 raft_consensus.cc:2229] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:15.441857 12176 raft_consensus.cc:2258] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:15.442677 12177 tablet_replica.cc:330] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: stopping tablet replica
I20260619 04:46:15.443194 12177 raft_consensus.cc:2229] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:15.443751 12177 raft_consensus.cc:2258] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:15.445936 12176 ts_tablet_manager.cc:1905] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:15.445936 12177 ts_tablet_manager.cc:1905] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:15.450456 12055 tablet_service.cc:1467] Processing CreateTablet for tablet 5c345be4f5824514b57a02a70ace1909 (DEFAULT_TABLE table=test-table [id=2f6e6ed60d6f45dfabfa11e855864aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:15.450968 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c345be4f5824514b57a02a70ace1909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:15.450927 11915 tablet_service.cc:1467] Processing CreateTablet for tablet 5c345be4f5824514b57a02a70ace1909 (DEFAULT_TABLE table=test-table [id=2f6e6ed60d6f45dfabfa11e855864aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:15.451330 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c345be4f5824514b57a02a70ace1909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:15.451491 11776 tablet_service.cc:1467] Processing CreateTablet for tablet 5c345be4f5824514b57a02a70ace1909 (DEFAULT_TABLE table=test-table [id=2f6e6ed60d6f45dfabfa11e855864aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:15.452011 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c345be4f5824514b57a02a70ace1909. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:15.453601 12177 ts_tablet_manager.cc:1918] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:15.453756 12177 log.cc:1196] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/07c618720751425eb07d4668167125ec
I20260619 04:46:15.454262 12177 ts_tablet_manager.cc:1939] T 07c618720751425eb07d4668167125ec P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting consensus metadata
I20260619 04:46:15.456167 11625 catalog_manager.cc:4841] TS cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235): tablet 07c618720751425eb07d4668167125ec (table test-table [id=9100cb40bde94e778360dcce5504370d]) successfully deleted
I20260619 04:46:15.457418 12150 tablet_bootstrap.cc:492] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Bootstrap starting.
I20260619 04:46:15.457731 12151 tablet_bootstrap.cc:492] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: Bootstrap starting.
I20260619 04:46:15.458985 12150 tablet_bootstrap.cc:654] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:15.459890 12151 tablet_bootstrap.cc:654] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:15.461265 12152 tablet_bootstrap.cc:492] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Bootstrap starting.
I20260619 04:46:15.463320 12152 tablet_bootstrap.cc:654] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:15.466516 12152 tablet_bootstrap.cc:492] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: No bootstrap required, opened a new log
I20260619 04:46:15.466723 12152 ts_tablet_manager.cc:1397] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 04:46:15.468652 12152 raft_consensus.cc:348] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.469014 12152 raft_consensus.cc:374] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:15.469069 12151 tablet_bootstrap.cc:492] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: No bootstrap required, opened a new log
I20260619 04:46:15.469158 12152 raft_consensus.cc:729] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4ae7acc2eb4e97b3fcb8224e8281e9, State: Initialized, Role: FOLLOWER
I20260619 04:46:15.469334 12151 ts_tablet_manager.cc:1397] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20260619 04:46:15.469561 12176 ts_tablet_manager.cc:1918] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:15.469470 12152 consensus_queue.cc:260] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.469718 12176 log.cc:1196] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/07c618720751425eb07d4668167125ec
I20260619 04:46:15.470279 12176 ts_tablet_manager.cc:1939] T 07c618720751425eb07d4668167125ec P dda69be6a15746f9819322d73dec9a81: Deleting consensus metadata
I20260619 04:46:15.470489 12151 raft_consensus.cc:348] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.470746 12151 raft_consensus.cc:374] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:15.470885 12151 raft_consensus.cc:729] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda69be6a15746f9819322d73dec9a81, State: Initialized, Role: FOLLOWER
I20260619 04:46:15.471283 12151 consensus_queue.cc:260] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.472892 12151 ts_tablet_manager.cc:1428] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260619 04:46:15.473541 12152 ts_tablet_manager.cc:1428] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260619 04:46:15.474200 12178 tablet_replica.cc:330] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: stopping tablet replica
I20260619 04:46:15.474464 12150 tablet_bootstrap.cc:492] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: No bootstrap required, opened a new log
I20260619 04:46:15.474618 12150 ts_tablet_manager.cc:1397] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.002s
I20260619 04:46:15.474704 12178 raft_consensus.cc:2229] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:46:15.475091 11627 catalog_manager.cc:4841] TS dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985): tablet 07c618720751425eb07d4668167125ec (table test-table [id=9100cb40bde94e778360dcce5504370d]) successfully deleted
I20260619 04:46:15.475302 12178 raft_consensus.cc:2258] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:15.475467 12150 raft_consensus.cc:348] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.475701 12150 raft_consensus.cc:374] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:15.475806 12150 raft_consensus.cc:729] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0067a8d8d54d2b9b8d5ce4654c7da9, State: Initialized, Role: FOLLOWER
I20260619 04:46:15.476028 12150 consensus_queue.cc:260] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.476646 12150 ts_tablet_manager.cc:1428] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:46:15.477485 12178 ts_tablet_manager.cc:1905] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:15.485098 12178 ts_tablet_manager.cc:1918] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:15.485285 12178 log.cc:1196] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/07c618720751425eb07d4668167125ec
I20260619 04:46:15.485838 12178 ts_tablet_manager.cc:1939] T 07c618720751425eb07d4668167125ec P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting consensus metadata
I20260619 04:46:15.487586 11624 catalog_manager.cc:4841] TS ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961): tablet 07c618720751425eb07d4668167125ec (table test-table [id=9100cb40bde94e778360dcce5504370d]) successfully deleted
I20260619 04:46:15.549331 12164 raft_consensus.cc:482] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:15.549616 12164 raft_consensus.cc:504] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.550328 12164 leader_election.cc:288] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985), cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235)
I20260619 04:46:15.550951 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c345be4f5824514b57a02a70ace1909" candidate_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81" is_pre_election: true
I20260619 04:46:15.550973 12080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c345be4f5824514b57a02a70ace1909" candidate_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" is_pre_election: true
I20260619 04:46:15.551219 12080 raft_consensus.cc:2454] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in term 0.
I20260619 04:46:15.551239 11941 raft_consensus.cc:2454] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in term 0.
I20260619 04:46:15.551653 11730 leader_election.cc:302] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dda69be6a15746f9819322d73dec9a81, ed4ae7acc2eb4e97b3fcb8224e8281e9; no voters:
I20260619 04:46:15.551971 12164 raft_consensus.cc:2781] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:46:15.552078 12164 raft_consensus.cc:482] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:46:15.552220 12164 raft_consensus.cc:3037] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:15.555341 12164 raft_consensus.cc:504] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.555989 12164 leader_election.cc:288] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [CANDIDATE]: Term 1 election: Requested vote from peers dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985), cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235)
I20260619 04:46:15.556370 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c345be4f5824514b57a02a70ace1909" candidate_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81"
I20260619 04:46:15.556423 12080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c345be4f5824514b57a02a70ace1909" candidate_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9"
I20260619 04:46:15.556605 11941 raft_consensus.cc:3037] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:15.556620 12080 raft_consensus.cc:3037] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:15.559126 12080 raft_consensus.cc:2454] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in term 1.
I20260619 04:46:15.559513 11728 leader_election.cc:302] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, ed4ae7acc2eb4e97b3fcb8224e8281e9; no voters:
I20260619 04:46:15.559590 11941 raft_consensus.cc:2454] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed4ae7acc2eb4e97b3fcb8224e8281e9 in term 1.
I20260619 04:46:15.559779 12164 raft_consensus.cc:2781] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:15.560005 12164 raft_consensus.cc:686] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 LEADER]: Becoming Leader. State: Replica: ed4ae7acc2eb4e97b3fcb8224e8281e9, State: Running, Role: LEADER
I20260619 04:46:15.560269 12164 consensus_queue.cc:237] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.562887 11639 catalog_manager.cc:5495] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129). New cstate: current_term: 1 leader_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } health_report { overall_health: UNKNOWN } } }
Jun 19 04:46:15 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844372, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.190@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:15.640443 12188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.614824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.3.243.190:45455)
0619 04:46:15.622541 (+ 7717us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.622550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.622572 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:46:15.622634 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.622641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.622729 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.622853 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:15.624273 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.625694 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.627534 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.628371 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.628501 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.633058 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:15.633065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:15.633070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:15.633108 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:15.636877 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:15.637867 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:15.637873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:15.637877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:15.638016 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:15.638494 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:15.638499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:15.638503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:15.638870 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:15.639041 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.639634 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.640023 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":147,"server-negotiator.queue_time_us":7570,"thread_start_us":7452,"threads_started":1}
I20260619 04:46:15.640457 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.614801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45455 (local address 127.0.0.1:55284)
0619 04:46:15.621587 (+ 6786us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.621615 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.621809 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.622925 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.622931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.623812 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:15.624080 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.624093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.625860 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.625866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.627378 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.627393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.628564 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.628571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.628815 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.629642 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:15.629667 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:15.632831 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:15.637027 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:15.637044 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:15.637055 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:15.637718 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:15.638133 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:15.638140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:15.638145 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:15.638364 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:15.639029 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:15.639041 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:15.639275 (+ 234us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.639475 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.639853 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6567,"thread_start_us":7102,"threads_started":1}
I20260619 04:46:15.647200 11639 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:55284:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 04:46:15.648684 11639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:46:15.656457 11777 tablet_service.cc:1467] Processing CreateTablet for tablet 8429d2f70b474e54a43874af29338e60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dd9a3f64e1d4536b0e6fd306276714c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:46:15.656823 11915 tablet_service.cc:1467] Processing CreateTablet for tablet 8429d2f70b474e54a43874af29338e60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dd9a3f64e1d4536b0e6fd306276714c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:46:15.657166 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8429d2f70b474e54a43874af29338e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:15.657307 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8429d2f70b474e54a43874af29338e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:15.659732 12055 tablet_service.cc:1467] Processing CreateTablet for tablet 8429d2f70b474e54a43874af29338e60 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9dd9a3f64e1d4536b0e6fd306276714c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 04:46:15.660279 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8429d2f70b474e54a43874af29338e60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:15.668859 12152 tablet_bootstrap.cc:492] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Bootstrap starting.
I20260619 04:46:15.671087 12152 tablet_bootstrap.cc:654] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:15.672410 12151 tablet_bootstrap.cc:492] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: Bootstrap starting.
I20260619 04:46:15.674055 12151 tablet_bootstrap.cc:654] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:15.678848 12150 tablet_bootstrap.cc:492] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Bootstrap starting.
I20260619 04:46:15.680477 12150 tablet_bootstrap.cc:654] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:15.683514 12152 tablet_bootstrap.cc:492] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: No bootstrap required, opened a new log
I20260619 04:46:15.683678 12151 tablet_bootstrap.cc:492] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: No bootstrap required, opened a new log
I20260619 04:46:15.683696 12152 ts_tablet_manager.cc:1397] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20260619 04:46:15.683838 12151 ts_tablet_manager.cc:1397] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20260619 04:46:15.683959 12150 tablet_bootstrap.cc:492] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: No bootstrap required, opened a new log
I20260619 04:46:15.684217 12150 ts_tablet_manager.cc:1397] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 04:46:15.684513 12152 raft_consensus.cc:348] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.684525 12151 raft_consensus.cc:348] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.684767 12151 raft_consensus.cc:374] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:15.684777 12152 raft_consensus.cc:374] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:15.684854 12151 raft_consensus.cc:729] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda69be6a15746f9819322d73dec9a81, State: Initialized, Role: FOLLOWER
I20260619 04:46:15.684860 12152 raft_consensus.cc:729] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4ae7acc2eb4e97b3fcb8224e8281e9, State: Initialized, Role: FOLLOWER
I20260619 04:46:15.685083 12150 raft_consensus.cc:348] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.685128 12151 consensus_queue.cc:260] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.685395 12150 raft_consensus.cc:374] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:15.685493 12150 raft_consensus.cc:729] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0067a8d8d54d2b9b8d5ce4654c7da9, State: Initialized, Role: FOLLOWER
I20260619 04:46:15.685563 12158 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.685786 12151 ts_tablet_manager.cc:1428] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:46:15.685858 12158 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:46:15.685762 12150 consensus_queue.cc:260] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.686324 12156 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.686614 12156 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:46:15.687493 12152 consensus_queue.cc:260] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.687973 12164 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.689159 12164 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:46:15.690395 12150 ts_tablet_manager.cc:1428] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260619 04:46:15.693248 12152 ts_tablet_manager.cc:1428] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20260619 04:46:15.743140 12156 raft_consensus.cc:482] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:15.743400 12156 raft_consensus.cc:504] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.745332 12156 leader_election.cc:288] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961), dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985)
I20260619 04:46:15.766124 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.746592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:38125 (local address 127.3.243.130:35985)
0619 04:46:15.746787 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.746795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.757224 (+ 10429us) server_negotiation.cc:407] Connection header received
0619 04:46:15.757969 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.757976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.758063 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.758134 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:15.758938 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.760286 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.763307 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.764737 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.764918 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.765011 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.765633 (+ 622us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.765814 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:46:15.766819 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.746166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:39475 (local address 127.3.243.129:40961)
0619 04:46:15.746406 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.746414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.756927 (+ 10513us) server_negotiation.cc:407] Connection header received
0619 04:46:15.757522 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.757531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.757634 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.757716 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:15.758792 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.760221 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.763529 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.765252 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.765427 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.765548 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.766194 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.766397 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:46:15.767040 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.745763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:35985 (local address 127.3.243.131:38125)
0619 04:46:15.757100 (+ 11337us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.757125 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.757762 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.758339 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.758344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.758372 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:15.758739 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.758752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.760478 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.760483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.763092 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.763109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.765227 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.765235 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.765410 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.765454 (+ 44us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.766458 (+ 1004us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.766621 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11187,"mutex_wait_us":99,"thread_start_us":10668,"threads_started":1}
I20260619 04:46:15.768065 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429d2f70b474e54a43874af29338e60" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81" is_pre_election: true
I20260619 04:46:15.768332 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.745640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:40961 (local address 127.3.243.131:39475)
0619 04:46:15.756790 (+ 11150us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.756816 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.757277 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.757971 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.757977 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.758006 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:15.758518 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.758536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.760424 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.760431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.763275 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.763294 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.765753 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.765762 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.765982 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.766028 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.767560 (+ 1532us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.767748 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10963,"mutex_wait_us":129,"thread_start_us":11305,"threads_started":1}
I20260619 04:46:15.768373 11941 raft_consensus.cc:2454] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 0.
I20260619 04:46:15.769336 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429d2f70b474e54a43874af29338e60" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" is_pre_election: true
I20260619 04:46:15.769649 11802 raft_consensus.cc:2454] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 0.
I20260619 04:46:15.770017 12009 leader_election.cc:302] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, dda69be6a15746f9819322d73dec9a81; no voters:
I20260619 04:46:15.770584 12156 raft_consensus.cc:2781] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:46:15.770736 12156 raft_consensus.cc:482] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:46:15.770834 12156 raft_consensus.cc:3037] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:15.774451 12156 raft_consensus.cc:504] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.775810 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429d2f70b474e54a43874af29338e60" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9"
I20260619 04:46:15.775830 11941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8429d2f70b474e54a43874af29338e60" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81"
I20260619 04:46:15.776064 11941 raft_consensus.cc:3037] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:15.776065 11802 raft_consensus.cc:3037] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:15.779583 11802 raft_consensus.cc:2454] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 1.
I20260619 04:46:15.779596 11941 raft_consensus.cc:2454] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 1.
I20260619 04:46:15.780064 12009 leader_election.cc:302] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, dda69be6a15746f9819322d73dec9a81; no voters:
I20260619 04:46:15.780450 12156 leader_election.cc:288] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 election: Requested vote from peers ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961), dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985)
I20260619 04:46:15.780746 12156 raft_consensus.cc:2781] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:15.780951 12156 raft_consensus.cc:686] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 LEADER]: Becoming Leader. State: Replica: cb0067a8d8d54d2b9b8d5ce4654c7da9, State: Running, Role: LEADER
I20260619 04:46:15.781416 12156 consensus_queue.cc:237] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:15.792791 12195 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: TxnStatusTablet state changed. Reason: New leader cb0067a8d8d54d2b9b8d5ce4654c7da9. Latest consensus state: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.793229 12195 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:46:15.794772 12197 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 04:46:15.794737 11640 catalog_manager.cc:5495] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } health_report { overall_health: HEALTHY } } }
I20260619 04:46:15.795096 12197 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 04:46:15.844177 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.835223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:43235 (local address 127.0.0.1:50700)
0619 04:46:15.835836 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.835857 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.836021 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.837173 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.837178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.837192 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:46:15.837383 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.837395 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.838948 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.838956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.840613 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.840631 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.841816 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.841823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.842032 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.842135 (+ 103us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:46:15.843467 (+ 1332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:46:15.843509 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.843593 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.843777 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401}
I20260619 04:46:15.845274 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.836401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.3.243.131:43235)
0619 04:46:15.836582 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.836588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.836607 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:46:15.836714 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.836721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.836828 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.836982 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:46:15.837571 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.838783 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.840791 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.841465 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.841628 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.842408 (+ 780us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:46:15.843271 (+ 863us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:46:15.843408 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.844546 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.844939 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:46:15.850843 11941 raft_consensus.cc:1261] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Refusing update from remote peer cb0067a8d8d54d2b9b8d5ce4654c7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:15.851997 11802 raft_consensus.cc:1261] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Refusing update from remote peer cb0067a8d8d54d2b9b8d5ce4654c7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:15.853153 12156 consensus_queue.cc:1035] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:15.854664 12156 consensus_queue.cc:1035] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:15.860918 12158 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: TxnStatusTablet state changed. Reason: New leader cb0067a8d8d54d2b9b8d5ce4654c7da9. Latest consensus state: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.861225 12158 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:46:15.870092 12156 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.870426 12156 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:46:15.873692 12158 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.874011 12158 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P dda69be6a15746f9819322d73dec9a81: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:46:15.895130 12164 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: TxnStatusTablet state changed. Reason: New leader cb0067a8d8d54d2b9b8d5ce4654c7da9. Latest consensus state: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.895603 12164 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:46:15.899688 12195 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.899976 12195 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P cb0067a8d8d54d2b9b8d5ce4654c7da9: This TxnStatusTablet replica's current role is: LEADER
I20260619 04:46:15.902894 12164 tablet_replica.cc:439] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } } }
I20260619 04:46:15.903180 12164 tablet_replica.cc:442] T 8429d2f70b474e54a43874af29338e60 P ed4ae7acc2eb4e97b3fcb8224e8281e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 04:46:15.936388 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.918241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:37271 (local address 127.3.243.131:43235)
0619 04:46:15.918465 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:15.918472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:15.928935 (+ 10463us) server_negotiation.cc:407] Connection header received
0619 04:46:15.929334 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:15.929340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:15.929442 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:15.929522 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:46:15.930340 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.931554 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.933868 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.934569 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.934684 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.935455 (+ 771us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:46:15.935591 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:46:15.935658 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:15.935911 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 04:46:15.935999 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:46:15.937368 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:15.917756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:43235 (local address 127.3.243.129:37271)
0619 04:46:15.928779 (+ 11023us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:15.928810 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:15.929115 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:15.929745 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:15.929753 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:15.929766 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:46:15.930091 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.930101 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.931742 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.931750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.933684 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:15.933701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:15.934907 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:15.934913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:15.935135 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:15.935208 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:46:15.935753 (+ 545us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:46:15.935764 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:46:15.936798 (+ 1034us) client_negotiation.cc:240] Negotiation successful
0619 04:46:15.936938 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10791,"thread_start_us":14377,"threads_started":1}
I20260619 04:46:15.946316 12164 consensus_queue.cc:1035] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:15.957021 12080 raft_consensus.cc:1261] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Refusing update from remote peer ed4ae7acc2eb4e97b3fcb8224e8281e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:15.957760 12164 consensus_queue.cc:1035] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:16.035240 12149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:16.016554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:42487 (local address 127.3.243.129:40961)
0619 04:46:16.016739 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:16.016745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:16.028556 (+ 11811us) server_negotiation.cc:407] Connection header received
0619 04:46:16.028857 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:16.028864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:16.028955 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:16.029023 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:46:16.029756 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.030828 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.032964 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.033646 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.033737 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:16.034385 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:46:16.034523 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:46:16.034580 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:16.034817 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 04:46:16.034901 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:46:16.035985 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:16.013817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:40961 (local address 127.3.243.131:42487)
0619 04:46:16.028442 (+ 14625us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:16.028462 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:16.028675 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:16.029208 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:16.029213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:16.029226 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:46:16.029571 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:16.029583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.031006 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.031013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:16.032796 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:16.032812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.033907 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.033912 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:16.034101 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:16.034186 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:46:16.034667 (+ 481us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:46:16.034676 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 04:46:16.035430 (+ 754us) client_negotiation.cc:240] Negotiation successful
0619 04:46:16.035584 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14197,"thread_start_us":14275,"threads_started":1}
I20260619 04:46:16.101408 11639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:16.101684 11639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:16.105075 11639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a: Sending DeleteTablet for 3 replicas of tablet 5c345be4f5824514b57a02a70ace1909
I20260619 04:46:16.105989 11777 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c345be4f5824514b57a02a70ace1909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:36354
I20260619 04:46:16.106083 11915 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c345be4f5824514b57a02a70ace1909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:38720
I20260619 04:46:16.106771 12055 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c345be4f5824514b57a02a70ace1909 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:50698
I20260619 04:46:16.111922 12211 tablet_replica.cc:330] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: stopping tablet replica
I20260619 04:46:16.112282 12211 raft_consensus.cc:2229] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:46:16.112825 12211 raft_consensus.cc:2258] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:16.115243 12211 ts_tablet_manager.cc:1905] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:16.125097 12211 ts_tablet_manager.cc:1918] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:46:16.125702 12212 tablet_replica.cc:330] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: stopping tablet replica
I20260619 04:46:16.126008 12212 raft_consensus.cc:2229] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:16.126323 12212 raft_consensus.cc:2258] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:16.127285 12212 ts_tablet_manager.cc:1905] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:16.130894 12213 tablet_replica.cc:330] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: stopping tablet replica
I20260619 04:46:16.126616 12211 log.cc:1196] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/5c345be4f5824514b57a02a70ace1909
I20260619 04:46:16.131120 12213 raft_consensus.cc:2229] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:16.131562 12213 raft_consensus.cc:2258] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:16.131630 12211 ts_tablet_manager.cc:1939] T 5c345be4f5824514b57a02a70ace1909 P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting consensus metadata
I20260619 04:46:16.132828 11624 catalog_manager.cc:4841] TS ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961): tablet 5c345be4f5824514b57a02a70ace1909 (table test-table [id=2f6e6ed60d6f45dfabfa11e855864aa5]) successfully deleted
I20260619 04:46:16.133252 12213 ts_tablet_manager.cc:1905] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:16.133711 12212 ts_tablet_manager.cc:1918] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:46:16.133909 12212 log.cc:1196] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/5c345be4f5824514b57a02a70ace1909
I20260619 04:46:16.134404 12212 ts_tablet_manager.cc:1939] T 5c345be4f5824514b57a02a70ace1909 P dda69be6a15746f9819322d73dec9a81: Deleting consensus metadata
I20260619 04:46:16.136001 11627 catalog_manager.cc:4841] TS dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985): tablet 5c345be4f5824514b57a02a70ace1909 (table test-table [id=2f6e6ed60d6f45dfabfa11e855864aa5]) successfully deleted
I20260619 04:46:16.141754 12213 ts_tablet_manager.cc:1918] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:46:16.141978 12213 log.cc:1196] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/5c345be4f5824514b57a02a70ace1909
I20260619 04:46:16.142649 12213 ts_tablet_manager.cc:1939] T 5c345be4f5824514b57a02a70ace1909 P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting consensus metadata
I20260619 04:46:16.144248 11625 catalog_manager.cc:4841] TS cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235): tablet 5c345be4f5824514b57a02a70ace1909 (table test-table [id=2f6e6ed60d6f45dfabfa11e855864aa5]) successfully deleted
Jun 19 04:46:16 dist-test-slave-2qp8 krb5kdc[11595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 04:46:16 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:16 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 04:46:16.185791 12221 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:16.171829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.3.243.190:45455)
0619 04:46:16.172524 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:16.172531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:16.172860 (+ 329us) server_negotiation.cc:407] Connection header received
0619 04:46:16.173016 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:16.173021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:16.173098 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:16.173221 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:16.174471 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.175629 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.176940 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.177585 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.177710 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:16.185417 (+ 7707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55298: BlockingRecv error: recv got EOF from 127.0.0.1:55298 (error 108)
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":441,"threads_started":1}
Jun 19 04:46:16 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM
I20260619 04:46:16.205353 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:16.189429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55304 (local address 127.3.243.190:45455)
0619 04:46:16.189714 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:16.189721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:16.190055 (+ 334us) server_negotiation.cc:407] Connection header received
0619 04:46:16.190265 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:16.190271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:16.190371 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:16.190497 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:16.191621 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.192439 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.193646 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.194072 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.194195 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:16.199519 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:16.199525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:16.199529 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:16.199568 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:16.202502 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:16.203407 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:16.203411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:16.203414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:16.203557 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:16.203967 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:16.203971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:16.203973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:16.204306 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:16.204523 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:16.204893 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 04:46:16.205069 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:46:16.209424 11639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:46:16.210059 11639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:46:16.217437 11777 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2a4e57a90e40d09dee106be9566a4e (DEFAULT_TABLE table=test-table [id=e75d770022ae4f65a134a1a449bc128b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:16.217736 11915 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2a4e57a90e40d09dee106be9566a4e (DEFAULT_TABLE table=test-table [id=e75d770022ae4f65a134a1a449bc128b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:16.217923 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2a4e57a90e40d09dee106be9566a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:16.218154 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2a4e57a90e40d09dee106be9566a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:16.224714 12228 tablet_bootstrap.cc:492] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: Bootstrap starting.
I20260619 04:46:16.224862 12055 tablet_service.cc:1467] Processing CreateTablet for tablet 8d2a4e57a90e40d09dee106be9566a4e (DEFAULT_TABLE table=test-table [id=e75d770022ae4f65a134a1a449bc128b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:16.225363 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d2a4e57a90e40d09dee106be9566a4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:16.228914 12228 tablet_bootstrap.cc:654] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:16.231650 12227 tablet_bootstrap.cc:492] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: Bootstrap starting.
I20260619 04:46:16.233048 12229 tablet_bootstrap.cc:492] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: Bootstrap starting.
I20260619 04:46:16.233489 12227 tablet_bootstrap.cc:654] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:16.235321 12229 tablet_bootstrap.cc:654] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:16.236917 12228 tablet_bootstrap.cc:492] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: No bootstrap required, opened a new log
I20260619 04:46:16.237119 12228 ts_tablet_manager.cc:1397] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260619 04:46:16.237308 12227 tablet_bootstrap.cc:492] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: No bootstrap required, opened a new log
I20260619 04:46:16.237515 12227 ts_tablet_manager.cc:1397] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 04:46:16.237989 12228 raft_consensus.cc:348] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.238332 12228 raft_consensus.cc:374] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:16.238479 12228 raft_consensus.cc:729] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4ae7acc2eb4e97b3fcb8224e8281e9, State: Initialized, Role: FOLLOWER
I20260619 04:46:16.238548 12227 raft_consensus.cc:348] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.238900 12227 raft_consensus.cc:374] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:16.238837 12228 consensus_queue.cc:260] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.239048 12227 raft_consensus.cc:729] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda69be6a15746f9819322d73dec9a81, State: Initialized, Role: FOLLOWER
I20260619 04:46:16.239495 12228 ts_tablet_manager.cc:1428] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 04:46:16.239789 12227 consensus_queue.cc:260] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.240453 12227 ts_tablet_manager.cc:1428] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 04:46:16.246321 12229 tablet_bootstrap.cc:492] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: No bootstrap required, opened a new log
I20260619 04:46:16.246773 12229 ts_tablet_manager.cc:1397] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260619 04:46:16.247586 12229 raft_consensus.cc:348] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.247779 12229 raft_consensus.cc:374] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:16.247901 12229 raft_consensus.cc:729] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0067a8d8d54d2b9b8d5ce4654c7da9, State: Initialized, Role: FOLLOWER
I20260619 04:46:16.248214 12229 consensus_queue.cc:260] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.248765 12229 ts_tablet_manager.cc:1428] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 04:46:16.388330 12156 raft_consensus.cc:482] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:16.388551 12156 raft_consensus.cc:504] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.389328 12156 leader_election.cc:288] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961), dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985)
I20260619 04:46:16.390010 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2a4e57a90e40d09dee106be9566a4e" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81" is_pre_election: true
I20260619 04:46:16.389991 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2a4e57a90e40d09dee106be9566a4e" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" is_pre_election: true
I20260619 04:46:16.390255 11802 raft_consensus.cc:2454] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 0.
I20260619 04:46:16.390255 11940 raft_consensus.cc:2454] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 0.
I20260619 04:46:16.390715 12006 leader_election.cc:302] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, ed4ae7acc2eb4e97b3fcb8224e8281e9; no voters:
I20260619 04:46:16.391080 12156 raft_consensus.cc:2781] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:46:16.391181 12156 raft_consensus.cc:482] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:46:16.391290 12156 raft_consensus.cc:3037] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:16.393247 12156 raft_consensus.cc:504] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.393802 12156 leader_election.cc:288] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 election: Requested vote from peers ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961), dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985)
I20260619 04:46:16.394181 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2a4e57a90e40d09dee106be9566a4e" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9"
I20260619 04:46:16.394251 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d2a4e57a90e40d09dee106be9566a4e" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81"
I20260619 04:46:16.394398 11802 raft_consensus.cc:3037] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:16.394446 11940 raft_consensus.cc:3037] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:16.397457 11940 raft_consensus.cc:2454] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 1.
I20260619 04:46:16.397456 11802 raft_consensus.cc:2454] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 1.
I20260619 04:46:16.397867 12009 leader_election.cc:302] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, dda69be6a15746f9819322d73dec9a81; no voters:
I20260619 04:46:16.398152 12156 raft_consensus.cc:2781] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:16.398284 12156 raft_consensus.cc:686] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 LEADER]: Becoming Leader. State: Replica: cb0067a8d8d54d2b9b8d5ce4654c7da9, State: Running, Role: LEADER
I20260619 04:46:16.398550 12156 consensus_queue.cc:237] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.401232 11639 catalog_manager.cc:5495] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } health_report { overall_health: HEALTHY } } }
I20260619 04:46:16.426585 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:16.421256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43616 (local address 127.3.243.131:43235)
0619 04:46:16.421440 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:16.421445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:16.421532 (+ 87us) server_negotiation.cc:407] Connection header received
0619 04:46:16.421706 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:16.421712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:16.421790 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:16.421884 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:46:16.422387 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.423452 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.424799 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.425446 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.425563 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:16.425924 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:46:16.426046 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:46:16.426133 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:16.426244 (+ 111us) server_negotiation.cc:299] Negotiation successful
0619 04:46:16.426318 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:46:16.429484 11940 raft_consensus.cc:1261] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Refusing update from remote peer cb0067a8d8d54d2b9b8d5ce4654c7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:16.429543 11802 raft_consensus.cc:1261] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Refusing update from remote peer cb0067a8d8d54d2b9b8d5ce4654c7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:16.430135 12156 consensus_queue.cc:1035] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:16.430374 12195 consensus_queue.cc:1035] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:16.443784 11639 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:16.444078 11639 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:16.447414 11639 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a: Sending DeleteTablet for 3 replicas of tablet 8d2a4e57a90e40d09dee106be9566a4e
I20260619 04:46:16.448275 11915 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2a4e57a90e40d09dee106be9566a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:38720
I20260619 04:46:16.448408 11776 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2a4e57a90e40d09dee106be9566a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:36354
I20260619 04:46:16.448625 11777 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2a4e57a90e40d09dee106be9566a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:36354
I20260619 04:46:16.448701 12212 tablet_replica.cc:330] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: stopping tablet replica
I20260619 04:46:16.448895 12212 raft_consensus.cc:2229] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:16.449116 12212 raft_consensus.cc:2258] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:16.449364 12211 tablet_replica.cc:330] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: stopping tablet replica
I20260619 04:46:16.449548 12211 raft_consensus.cc:2229] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:16.449754 12211 raft_consensus.cc:2258] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:16.450071 11639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 04:46:16.450551 12055 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d2a4e57a90e40d09dee106be9566a4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:50698
W20260619 04:46:16.450675 11639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:46:16.450840 12213 tablet_replica.cc:330] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: stopping tablet replica
I20260619 04:46:16.451090 12213 raft_consensus.cc:2229] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:46:16.451442 12211 ts_tablet_manager.cc:1905] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:16.451584 12213 raft_consensus.cc:2258] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:16.452772 12213 ts_tablet_manager.cc:1905] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:16.457939 12211 ts_tablet_manager.cc:1918] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:16.458079 12211 log.cc:1196] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/8d2a4e57a90e40d09dee106be9566a4e
I20260619 04:46:16.458299 11777 tablet_service.cc:1467] Processing CreateTablet for tablet 848ee37ffc3448b0a47c51ad9a3c10fe (DEFAULT_TABLE table=test-table [id=4b154284e94a4c25b2e335a4d11c10f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:16.458509 12211 ts_tablet_manager.cc:1939] T 8d2a4e57a90e40d09dee106be9566a4e P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting consensus metadata
I20260619 04:46:16.458693 11777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848ee37ffc3448b0a47c51ad9a3c10fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:16.459275 12213 ts_tablet_manager.cc:1918] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:16.459408 12213 log.cc:1196] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/8d2a4e57a90e40d09dee106be9566a4e
I20260619 04:46:16.459579 11624 catalog_manager.cc:4841] TS ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961): tablet 8d2a4e57a90e40d09dee106be9566a4e (table test-table [id=e75d770022ae4f65a134a1a449bc128b]) successfully deleted
I20260619 04:46:16.459786 12213 ts_tablet_manager.cc:1939] T 8d2a4e57a90e40d09dee106be9566a4e P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting consensus metadata
I20260619 04:46:16.460698 12212 ts_tablet_manager.cc:1905] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:16.461328 11915 tablet_service.cc:1467] Processing CreateTablet for tablet 848ee37ffc3448b0a47c51ad9a3c10fe (DEFAULT_TABLE table=test-table [id=4b154284e94a4c25b2e335a4d11c10f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:16.461725 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848ee37ffc3448b0a47c51ad9a3c10fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:16.465440 12228 tablet_bootstrap.cc:492] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: Bootstrap starting.
I20260619 04:46:16.466126 11625 catalog_manager.cc:4841] TS cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235): tablet 8d2a4e57a90e40d09dee106be9566a4e (table test-table [id=e75d770022ae4f65a134a1a449bc128b]) successfully deleted
I20260619 04:46:16.466622 12055 tablet_service.cc:1467] Processing CreateTablet for tablet 848ee37ffc3448b0a47c51ad9a3c10fe (DEFAULT_TABLE table=test-table [id=4b154284e94a4c25b2e335a4d11c10f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:16.467025 12055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 848ee37ffc3448b0a47c51ad9a3c10fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:16.467078 12228 tablet_bootstrap.cc:654] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:16.470698 12212 ts_tablet_manager.cc:1918] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:16.470849 12212 log.cc:1196] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/8d2a4e57a90e40d09dee106be9566a4e
I20260619 04:46:16.471180 12227 tablet_bootstrap.cc:492] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: Bootstrap starting.
I20260619 04:46:16.471284 12212 ts_tablet_manager.cc:1939] T 8d2a4e57a90e40d09dee106be9566a4e P dda69be6a15746f9819322d73dec9a81: Deleting consensus metadata
I20260619 04:46:16.472692 12227 tablet_bootstrap.cc:654] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:16.473125 12228 tablet_bootstrap.cc:492] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: No bootstrap required, opened a new log
I20260619 04:46:16.473278 12228 ts_tablet_manager.cc:1397] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260619 04:46:16.474052 12228 raft_consensus.cc:348] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
W20260619 04:46:16.474164 11624 catalog_manager.cc:4808] TS ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961): delete failed for tablet 8d2a4e57a90e40d09dee106be9566a4e because the tablet was not found. No further retry: Not found: Tablet not found: 8d2a4e57a90e40d09dee106be9566a4e
I20260619 04:46:16.474293 12228 raft_consensus.cc:374] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:16.474363 12228 raft_consensus.cc:729] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed4ae7acc2eb4e97b3fcb8224e8281e9, State: Initialized, Role: FOLLOWER
I20260619 04:46:16.474689 11627 catalog_manager.cc:4841] TS dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985): tablet 8d2a4e57a90e40d09dee106be9566a4e (table test-table [id=e75d770022ae4f65a134a1a449bc128b]) successfully deleted
I20260619 04:46:16.474604 12228 consensus_queue.cc:260] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.475200 12228 ts_tablet_manager.cc:1428] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 04:46:16.477478 12227 tablet_bootstrap.cc:492] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: No bootstrap required, opened a new log
I20260619 04:46:16.477643 12227 ts_tablet_manager.cc:1397] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260619 04:46:16.478413 12227 raft_consensus.cc:348] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.478689 12227 raft_consensus.cc:374] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:16.478775 12227 raft_consensus.cc:729] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda69be6a15746f9819322d73dec9a81, State: Initialized, Role: FOLLOWER
I20260619 04:46:16.479017 12227 consensus_queue.cc:260] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.479519 12227 ts_tablet_manager.cc:1428] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 04:46:16.480616 12229 tablet_bootstrap.cc:492] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: Bootstrap starting.
I20260619 04:46:16.482512 12229 tablet_bootstrap.cc:654] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:16.485081 12229 tablet_bootstrap.cc:492] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: No bootstrap required, opened a new log
I20260619 04:46:16.485268 12229 ts_tablet_manager.cc:1397] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 04:46:16.486042 12229 raft_consensus.cc:348] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.486310 12229 raft_consensus.cc:374] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:16.486398 12229 raft_consensus.cc:729] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb0067a8d8d54d2b9b8d5ce4654c7da9, State: Initialized, Role: FOLLOWER
I20260619 04:46:16.486670 12229 consensus_queue.cc:260] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.487267 12229 ts_tablet_manager.cc:1428] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 04:46:16.737509 12156 raft_consensus.cc:482] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:16.737771 12156 raft_consensus.cc:504] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.738520 12156 leader_election.cc:288] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961), dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985)
I20260619 04:46:16.739064 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848ee37ffc3448b0a47c51ad9a3c10fe" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" is_pre_election: true
I20260619 04:46:16.739089 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848ee37ffc3448b0a47c51ad9a3c10fe" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81" is_pre_election: true
I20260619 04:46:16.739320 11940 raft_consensus.cc:2454] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 0.
I20260619 04:46:16.739320 11802 raft_consensus.cc:2454] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 0.
I20260619 04:46:16.739742 12006 leader_election.cc:302] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, ed4ae7acc2eb4e97b3fcb8224e8281e9; no voters:
I20260619 04:46:16.740062 12156 raft_consensus.cc:2781] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:46:16.740207 12156 raft_consensus.cc:482] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:46:16.740350 12156 raft_consensus.cc:3037] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:16.742760 12156 raft_consensus.cc:504] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.743330 12156 leader_election.cc:288] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 election: Requested vote from peers ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961), dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985)
I20260619 04:46:16.743744 11802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848ee37ffc3448b0a47c51ad9a3c10fe" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9"
I20260619 04:46:16.743961 11802 raft_consensus.cc:3037] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:16.743893 11940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "848ee37ffc3448b0a47c51ad9a3c10fe" candidate_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dda69be6a15746f9819322d73dec9a81"
I20260619 04:46:16.744134 11940 raft_consensus.cc:3037] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:16.746876 11802 raft_consensus.cc:2454] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 1.
I20260619 04:46:16.746948 11940 raft_consensus.cc:2454] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb0067a8d8d54d2b9b8d5ce4654c7da9 in term 1.
I20260619 04:46:16.747308 12006 leader_election.cc:302] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb0067a8d8d54d2b9b8d5ce4654c7da9, ed4ae7acc2eb4e97b3fcb8224e8281e9; no voters:
I20260619 04:46:16.747660 12156 raft_consensus.cc:2781] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:16.747857 12156 raft_consensus.cc:686] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 LEADER]: Becoming Leader. State: Replica: cb0067a8d8d54d2b9b8d5ce4654c7da9, State: Running, Role: LEADER
I20260619 04:46:16.748200 12156 consensus_queue.cc:237] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } }
I20260619 04:46:16.751349 11638 catalog_manager.cc:5495] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb0067a8d8d54d2b9b8d5ce4654c7da9" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 43235 } health_report { overall_health: HEALTHY } } }
I20260619 04:46:16.802116 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:16.796627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:54741 (local address 127.3.243.131:43235)
0619 04:46:16.796838 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:16.796845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:16.797427 (+ 582us) server_negotiation.cc:407] Connection header received
0619 04:46:16.797605 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:16.797610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:16.797681 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:16.797771 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:46:16.798259 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.798981 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.800423 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.800909 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.801039 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:16.801383 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:46:16.801501 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:46:16.801596 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:16.801747 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 04:46:16.801809 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 04:46:16.802239 12245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:16.796689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:43235 (local address 127.3.243.131:54741)
0619 04:46:16.797315 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:16.797332 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:16.797497 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:16.797790 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:16.797794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:16.797807 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 04:46:16.798112 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:16.798120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.799108 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.799113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:16.800299 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:16.800310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.801080 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.801087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:16.801213 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:16.801241 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 04:46:16.801614 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 04:46:16.801626 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 04:46:16.801722 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 04:46:16.801856 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":3,"thread_start_us":357,"threads_started":1}
I20260619 04:46:16.810582 11802 raft_consensus.cc:1261] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Refusing update from remote peer cb0067a8d8d54d2b9b8d5ce4654c7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:16.810770 11940 raft_consensus.cc:1261] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Refusing update from remote peer cb0067a8d8d54d2b9b8d5ce4654c7da9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:16.811254 12156 consensus_queue.cc:1035] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed4ae7acc2eb4e97b3fcb8224e8281e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:16.811544 12195 consensus_queue.cc:1035] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dda69be6a15746f9819322d73dec9a81" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 35985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:16.862985 11638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:16.863262 11638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:16.866057 11638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b93babe93aba4b2a9d4eb6af7a2e991a: Sending DeleteTablet for 3 replicas of tablet 848ee37ffc3448b0a47c51ad9a3c10fe
I20260619 04:46:16.866945 11777 tablet_service.cc:1514] Processing DeleteTablet for tablet 848ee37ffc3448b0a47c51ad9a3c10fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:36354
I20260619 04:46:16.867077 11915 tablet_service.cc:1514] Processing DeleteTablet for tablet 848ee37ffc3448b0a47c51ad9a3c10fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:38720
I20260619 04:46:16.867069 12054 tablet_service.cc:1514] Processing DeleteTablet for tablet 848ee37ffc3448b0a47c51ad9a3c10fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:16 UTC) from {username='oryx', principal='oryx/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:50698
I20260619 04:46:16.867304 12211 tablet_replica.cc:330] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: stopping tablet replica
I20260619 04:46:16.867369 12212 tablet_replica.cc:330] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: stopping tablet replica
I20260619 04:46:16.867370 12213 tablet_replica.cc:330] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: stopping tablet replica
I20260619 04:46:16.867512 12211 raft_consensus.cc:2229] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:16.867539 12212 raft_consensus.cc:2229] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:16.867540 12213 raft_consensus.cc:2229] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:46:16.867733 12212 raft_consensus.cc:2258] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:16.867729 12211 raft_consensus.cc:2258] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:16.867864 12213 raft_consensus.cc:2258] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:16.868932 12213 ts_tablet_manager.cc:1905] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:16.869038 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 11711
I20260619 04:46:16.869066 12211 ts_tablet_manager.cc:1905] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:16.869285 12212 ts_tablet_manager.cc:1905] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:16.875597 12213 ts_tablet_manager.cc:1918] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:46:16.875751 12213 log.cc:1196] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/848ee37ffc3448b0a47c51ad9a3c10fe
I20260619 04:46:16.876044 12211 ts_tablet_manager.cc:1918] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:46:16.876222 12213 ts_tablet_manager.cc:1939] T 848ee37ffc3448b0a47c51ad9a3c10fe P cb0067a8d8d54d2b9b8d5ce4654c7da9: Deleting consensus metadata
I20260619 04:46:16.876258 12211 log.cc:1196] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/848ee37ffc3448b0a47c51ad9a3c10fe
I20260619 04:46:16.876770 12211 ts_tablet_manager.cc:1939] T 848ee37ffc3448b0a47c51ad9a3c10fe P ed4ae7acc2eb4e97b3fcb8224e8281e9: Deleting consensus metadata
I20260619 04:46:16.877372 11625 catalog_manager.cc:4841] TS cb0067a8d8d54d2b9b8d5ce4654c7da9 (127.3.243.131:43235): tablet 848ee37ffc3448b0a47c51ad9a3c10fe (table test-table [id=4b154284e94a4c25b2e335a4d11c10f6]) successfully deleted
I20260619 04:46:16.878495 11624 catalog_manager.cc:4841] TS ed4ae7acc2eb4e97b3fcb8224e8281e9 (127.3.243.129:40961): tablet 848ee37ffc3448b0a47c51ad9a3c10fe (table test-table [id=4b154284e94a4c25b2e335a4d11c10f6]) successfully deleted
I20260619 04:46:16.878696 12212 ts_tablet_manager.cc:1918] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 04:46:16.878881 12212 log.cc:1196] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/848ee37ffc3448b0a47c51ad9a3c10fe
I20260619 04:46:16.879288 12212 ts_tablet_manager.cc:1939] T 848ee37ffc3448b0a47c51ad9a3c10fe P dda69be6a15746f9819322d73dec9a81: Deleting consensus metadata
I20260619 04:46:16.880329 11627 catalog_manager.cc:4841] TS dda69be6a15746f9819322d73dec9a81 (127.3.243.130:35985): tablet 848ee37ffc3448b0a47c51ad9a3c10fe (table test-table [id=4b154284e94a4c25b2e335a4d11c10f6]) successfully deleted
Jun 19 04:46:16 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.129@KRBTEST.COM
I20260619 04:46:16.892879 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:16.872161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.3.243.129:40961)
0619 04:46:16.877519 (+ 5358us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:16.877525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:16.878051 (+ 526us) server_negotiation.cc:407] Connection header received
0619 04:46:16.878249 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:16.878255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:16.878364 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:16.878499 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:16.879969 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.880957 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.882208 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:16.882594 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:16.882695 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:16.886956 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:16.886971 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:16.886993 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:16.887023 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:16.890193 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:16.890816 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:16.890825 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:16.890829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:16.890956 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:16.891332 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:16.891337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:16.891340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:16.891691 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:16.892027 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:16.892315 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 04:46:16.892532 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5198,"thread_start_us":5087,"threads_started":1}
I20260619 04:46:16.893558 11842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:17.021090 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 11711
I20260619 04:46:17.045296 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 11851
Jun 19 04:46:17 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.130@KRBTEST.COM
I20260619 04:46:17.059394 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:17.046085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.3.243.130:35985)
0619 04:46:17.046634 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:17.046638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:17.046653 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:46:17.046708 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:17.046712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:17.046791 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:17.046888 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:17.048096 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:17.048887 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:17.050035 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:17.050391 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:17.050475 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:17.054518 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:17.054531 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:17.054559 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:17.054590 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:17.057233 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:17.057809 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:17.057816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:17.057819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:17.057931 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:17.058204 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:17.058206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:17.058208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:17.058532 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:17.058770 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:17.058996 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 04:46:17.059147 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":350,"threads_started":1}
I20260619 04:46:17.060032 11981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:17.158318 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 11851
I20260619 04:46:17.176638 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 11990
Jun 19 04:46:17 dist-test-slave-2qp8 krb5kdc[11595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.243.131@KRBTEST.COM
I20260619 04:46:17.191536 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:17.177571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.3.243.131:43235)
0619 04:46:17.177744 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:17.177749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:17.177766 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:46:17.177826 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:17.177829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:17.177890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:17.177973 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:17.179175 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:17.179820 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:17.181024 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:17.181436 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:17.181528 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:17.185991 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:17.186007 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:17.186024 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:17.186055 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:17.189138 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:17.189734 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:17.189758 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:17.189764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:17.189899 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:17.190192 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:17.190196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:17.190198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:17.190605 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:17.190828 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:17.191017 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 04:46:17.191166 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:46:17.192296 12120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:17.337596 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 11990
I20260619 04:46:17.362485 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 11608
I20260619 04:46:17.363672 11680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:17.480377 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 11608
2026-06-19T04:46:17Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6971 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:46:17 dist-test-slave-2qp8 krb5kdc[12259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:46:17 dist-test-slave-2qp8 krb5kdc[12259](info): set up 2 sockets
Jun 19 04:46:17 dist-test-slave-2qp8 krb5kdc[12259](info): commencing operation
krb5kdc: starting...
W20260619 04:46:18.589211 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:46:18 dist-test-slave-2qp8 krb5kdc[12259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:46:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:46:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:46:18.801226 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:40401
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:40401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:19.117883 12272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:19.118153 12272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:19.118261 12272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:19.125324 12272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:19.125447 12272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:19.125532 12272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:19.125646 12272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:19.125737 12272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:19.134904 12272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:40401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:40401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:19.136138 12272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:19.137434 12272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:19.149000 12278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:19.148972 12280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:19.149034 12277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:19.149646 12272 server_base.cc:1034] running on GCE node
I20260619 04:46:19.150295 12272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:19.151551 12272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:19.152745 12272 hybrid_clock.cc:648] HybridClock initialized: now 1781844379152699 us; error 37 us; skew 500 ppm
I20260619 04:46:19.153035 12272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T04:46:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1736 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:46:19 dist-test-slave-2qp8 krb5kdc[12286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:46:19 dist-test-slave-2qp8 krb5kdc[12286](info): set up 2 sockets
Jun 19 04:46:19 dist-test-slave-2qp8 krb5kdc[12286](info): commencing operation
krb5kdc: starting...
W20260619 04:46:20.301137 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:46:20 dist-test-slave-2qp8 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:46:20.523393 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44025
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:20.838826 12299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:20.839149 12299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:20.839308 12299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:20.846318 12299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:20.846469 12299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:20.846581 12299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:20.846680 12299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:20.846758 12299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:20.856204 12299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:20.857565 12299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:20.858870 12299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:20.870524 12305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:20.870615 12304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:20.870679 12307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:20.871663 12299 server_base.cc:1034] running on GCE node
I20260619 04:46:20.872579 12299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:20.873993 12299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:20.875222 12299 hybrid_clock.cc:648] HybridClock initialized: now 1781844380875182 us; error 60 us; skew 500 ppm
I20260619 04:46:20.875578 12299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:20 dist-test-slave-2qp8 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844380, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:20.882722 12299 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:46:20.884443 12299 webserver.cc:460] Webserver started at http://127.3.243.190:40169/ using document root <none> and password file <none>
I20260619 04:46:20.885155 12299 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:20.885301 12299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:20.885624 12299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:20.888718 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "1b15ba73bb98431da6d308479f5896f4"
format_stamp: "Formatted at 2026-06-19 04:46:20 on dist-test-slave-2qp8"
server_key: "520f34686831a634470bc3d31c5855b3"
server_key_iv: "71855425bc1d20535adc1f8d51eb8b3d"
server_key_version: "encryptionkey@0"
I20260619 04:46:20.889514 12299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "1b15ba73bb98431da6d308479f5896f4"
format_stamp: "Formatted at 2026-06-19 04:46:20 on dist-test-slave-2qp8"
server_key: "520f34686831a634470bc3d31c5855b3"
server_key_iv: "71855425bc1d20535adc1f8d51eb8b3d"
server_key_version: "encryptionkey@0"
I20260619 04:46:20.894634 12299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 04:46:20.898010 12314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:20.899315 12299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:46:20.899497 12299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "1b15ba73bb98431da6d308479f5896f4"
format_stamp: "Formatted at 2026-06-19 04:46:20 on dist-test-slave-2qp8"
server_key: "520f34686831a634470bc3d31c5855b3"
server_key_iv: "71855425bc1d20535adc1f8d51eb8b3d"
server_key_version: "encryptionkey@0"
I20260619 04:46:20.899685 12299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:20.918329 12299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:20.922935 12299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:20.923254 12299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:20.943159 12299 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44025
I20260619 04:46:20.943187 12365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44025 every 8 connection(s)
I20260619 04:46:20.944799 12299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:46:20.948653 12366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:20.954066 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 12299
I20260619 04:46:20.954286 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:46:20.954901 4046 external_mini_cluster.cc:1428] Setting key 78251e42421b8c1e6d21e9f936727f99
I20260619 04:46:20.959645 12366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: Bootstrap starting.
I20260619 04:46:20.963450 12366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:20.964805 12366 log.cc:824] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:20.968288 12366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: No bootstrap required, opened a new log
Jun 19 04:46:20 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:46:20.979293 12366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } }
I20260619 04:46:20.979761 12366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:20.979882 12366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b15ba73bb98431da6d308479f5896f4, State: Initialized, Role: FOLLOWER
I20260619 04:46:20.980170 12369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:20.956870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42296 (local address 127.3.243.190:44025)
0619 04:46:20.957722 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:20.957734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:20.957773 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:46:20.959461 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:20.959484 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:20.959918 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:20.960291 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:20.962205 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:20.963521 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:20.965356 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:20.966401 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:20.966543 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:20.971243 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:20.971252 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:20.971272 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:20.971351 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:20.975539 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:20.976270 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:20.976279 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:20.976287 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:20.976499 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:20.976847 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:20.976852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:20.976855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:20.977646 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:20.977937 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:20.978267 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 04:46:20.978712 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":410,"threads_started":1}
I20260619 04:46:20.980670 12366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } }
I20260619 04:46:20.980926 12366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:46:20.981060 12366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:46:20.981242 12366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:20.983532 12366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } }
I20260619 04:46:20.984061 12366 leader_election.cc:302] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b15ba73bb98431da6d308479f5896f4; no voters:
I20260619 04:46:20.984658 12366 leader_election.cc:288] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:46:20.984805 12371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:20.985471 12371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 1 LEADER]: Becoming Leader. State: Replica: 1b15ba73bb98431da6d308479f5896f4, State: Running, Role: LEADER
I20260619 04:46:20.986009 12371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } }
I20260619 04:46:20.986743 12366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:46:20.988739 12373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b15ba73bb98431da6d308479f5896f4. Latest consensus state: current_term: 1 leader_uuid: "1b15ba73bb98431da6d308479f5896f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } } }
I20260619 04:46:20.988749 12372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b15ba73bb98431da6d308479f5896f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } } }
I20260619 04:46:20.989136 12373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:20.989208 12372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:20.990118 12377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:46:20.996932 12377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:46:21.007870 12377 catalog_manager.cc:1347] Generated new cluster ID: fe798862b3a3483a945df11051defbfe
I20260619 04:46:21.008003 12377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:46:21.025148 12377 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:46:21.027022 12377 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:46:21.052732 12377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: Generated new TSK 0
I20260619 04:46:21.053853 12377 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:46:21.118127 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44025
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:21.452889 12392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:21.453217 12392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:21.453398 12392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:21.460664 12392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:21.460816 12392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:21.461032 12392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:46:21.471007 12392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:21.472504 12392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:21.473749 12392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:21.486968 12400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:21.487036 12398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:21.487183 12397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:21.487578 12392 server_base.cc:1034] running on GCE node
I20260619 04:46:21.488386 12392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:21.489401 12392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:21.490639 12392 hybrid_clock.cc:648] HybridClock initialized: now 1781844381490583 us; error 50 us; skew 500 ppm
I20260619 04:46:21.490960 12392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:21 dist-test-slave-2qp8 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:21.494192 12392 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:46:21.495743 12392 webserver.cc:460] Webserver started at http://127.3.243.129:45713/ using document root <none> and password file <none>
I20260619 04:46:21.496497 12392 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:21.496649 12392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:21.496961 12392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:21.499962 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "afb832b9146b4b15a12ca79478898d05"
format_stamp: "Formatted at 2026-06-19 04:46:21 on dist-test-slave-2qp8"
server_key: "a96b39c26030b24c362cf88117216122"
server_key_iv: "31cbd90aa4affcb76e2e02e640e3966b"
server_key_version: "encryptionkey@0"
I20260619 04:46:21.500787 12392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "afb832b9146b4b15a12ca79478898d05"
format_stamp: "Formatted at 2026-06-19 04:46:21 on dist-test-slave-2qp8"
server_key: "a96b39c26030b24c362cf88117216122"
server_key_iv: "31cbd90aa4affcb76e2e02e640e3966b"
server_key_version: "encryptionkey@0"
I20260619 04:46:21.505857 12392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:46:21.509107 12407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:21.510500 12392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:46:21.510711 12392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "afb832b9146b4b15a12ca79478898d05"
format_stamp: "Formatted at 2026-06-19 04:46:21 on dist-test-slave-2qp8"
server_key: "a96b39c26030b24c362cf88117216122"
server_key_iv: "31cbd90aa4affcb76e2e02e640e3966b"
server_key_version: "encryptionkey@0"
I20260619 04:46:21.510931 12392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:21.528007 12392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:21.531726 12392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:21.532021 12392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:21.532925 12392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:21.534236 12392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:21.534343 12392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:21.534508 12392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:21.534581 12392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:21.568269 12392 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:33653
I20260619 04:46:21.568365 12519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:33653 every 8 connection(s)
I20260619 04:46:21.569525 12392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:46:21.573166 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 12392
I20260619 04:46:21.573377 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:46:21.574141 4046 external_mini_cluster.cc:1428] Setting key 834113e84a1a98661c06d2ab3d0b4b08
Jun 19 04:46:21 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:46:21.589665 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:21.572090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:42125 (local address 127.3.243.190:44025)
0619 04:46:21.572594 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:21.572599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:21.573684 (+ 1085us) server_negotiation.cc:407] Connection header received
0619 04:46:21.575699 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:21.575704 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:21.575791 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:21.575915 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:21.577400 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:21.578122 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:21.579629 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:21.580034 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:21.580194 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:21.583852 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:21.583858 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:21.583863 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:21.583904 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:21.586620 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:21.587335 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:21.587348 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:21.587351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:21.587465 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:21.587888 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:21.587893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:21.587896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:21.588241 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:21.588411 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:21.589097 (+ 686us) server_negotiation.cc:299] Negotiation successful
0619 04:46:21.589310 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":246,"threads_started":1}
I20260619 04:46:21.590633 12523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:21.572619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:42125)
0619 04:46:21.573510 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:21.573566 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:21.575450 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:21.576208 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:21.576217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:21.576706 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:21.577244 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:21.577260 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:21.578272 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:21.578277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:21.579480 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:21.579492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:21.580276 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:21.580285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:21.580520 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:21.581189 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:21.581214 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:21.583647 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:21.586752 (+ 3105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:21.586760 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:21.586776 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:21.587194 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:21.587585 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:21.587589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:21.587593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:21.587771 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:21.588380 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:21.588387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:21.588736 (+ 349us) client_negotiation.cc:769] Sending connection context
0619 04:46:21.588989 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 04:46:21.589494 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":311,"threads_started":1}
I20260619 04:46:21.592445 12520 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44025
I20260619 04:46:21.592801 12520 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:21.593744 12520 heartbeater.cc:511] Master 127.3.243.190:44025 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:46:21.596005 12331 ts_manager.cc:194] Registered new tserver with Master: afb832b9146b4b15a12ca79478898d05 (127.3.243.129:33653)
I20260619 04:46:21.597445 12331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:42125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:46:21.623996 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44025
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:21.956152 12526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:21.956511 12526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:21.956663 12526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:21.968014 12526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:21.968230 12526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:21.968386 12526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:46:21.978715 12526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:21.979996 12526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:21.981436 12526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:21.994843 12531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:21.994884 12532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:21.994903 12534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:21.996003 12526 server_base.cc:1034] running on GCE node
I20260619 04:46:21.996695 12526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:21.997692 12526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:21.998941 12526 hybrid_clock.cc:648] HybridClock initialized: now 1781844381998872 us; error 53 us; skew 500 ppm
I20260619 04:46:21.999295 12526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:21 dist-test-slave-2qp8 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:22.002519 12526 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:46:22.004042 12526 webserver.cc:460] Webserver started at http://127.3.243.130:43895/ using document root <none> and password file <none>
I20260619 04:46:22.004791 12526 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:22.004895 12526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:22.005149 12526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:22.007992 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "7192b0ba7d3f42e9b9463152e2fb4fca"
format_stamp: "Formatted at 2026-06-19 04:46:22 on dist-test-slave-2qp8"
server_key: "f15e547802b493c52fdf9c95f70de5b5"
server_key_iv: "ff8a446647ef2b33215f5239f38ba87f"
server_key_version: "encryptionkey@0"
I20260619 04:46:22.008910 12526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "7192b0ba7d3f42e9b9463152e2fb4fca"
format_stamp: "Formatted at 2026-06-19 04:46:22 on dist-test-slave-2qp8"
server_key: "f15e547802b493c52fdf9c95f70de5b5"
server_key_iv: "ff8a446647ef2b33215f5239f38ba87f"
server_key_version: "encryptionkey@0"
I20260619 04:46:22.013927 12526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:46:22.017612 12541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:22.018982 12526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:46:22.019114 12526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "7192b0ba7d3f42e9b9463152e2fb4fca"
format_stamp: "Formatted at 2026-06-19 04:46:22 on dist-test-slave-2qp8"
server_key: "f15e547802b493c52fdf9c95f70de5b5"
server_key_iv: "ff8a446647ef2b33215f5239f38ba87f"
server_key_version: "encryptionkey@0"
I20260619 04:46:22.019279 12526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:22.032578 12526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:22.036527 12526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:22.036768 12526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:22.037657 12526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:22.038872 12526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:22.038940 12526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:22.039019 12526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:22.039083 12526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:22.074132 12526 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:38027
I20260619 04:46:22.074225 12653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:38027 every 8 connection(s)
I20260619 04:46:22.075459 12526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:46:22.084735 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 12526
I20260619 04:46:22.084914 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:46:22.085458 4046 external_mini_cluster.cc:1428] Setting key db747e52289eb9ef05f5b6bfdd27cf9f
Jun 19 04:46:22 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844381, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:46:22.097957 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:22.078061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:41301 (local address 127.3.243.190:44025)
0619 04:46:22.078231 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:22.078236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:22.079472 (+ 1236us) server_negotiation.cc:407] Connection header received
0619 04:46:22.081285 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:22.081291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:22.081403 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:22.081532 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:22.082985 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.083696 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.085851 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.086429 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.086567 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:22.090412 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:22.090421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:22.090428 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:22.090472 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:22.094220 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.095107 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.095115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.095121 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.095288 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.095905 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.095910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.095912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.096351 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:22.096551 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:22.097449 (+ 898us) server_negotiation.cc:299] Negotiation successful
0619 04:46:22.097646 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:46:22.099392 12656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:22.078581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:41301)
0619 04:46:22.079308 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:22.079344 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:22.081035 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:22.081757 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:22.081765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:22.082267 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:22.082817 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:22.082834 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.083837 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.083849 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:22.085703 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:22.085729 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.086681 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.086688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:22.086978 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:22.087708 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:22.087739 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:22.090183 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:22.094528 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:22.094535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:22.094547 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:22.094971 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:22.095439 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:22.095445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:22.095450 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:22.095717 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:22.096525 (+ 808us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:22.096536 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:22.097014 (+ 478us) client_negotiation.cc:769] Sending connection context
0619 04:46:22.097306 (+ 292us) client_negotiation.cc:240] Negotiation successful
0619 04:46:22.097918 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":267,"threads_started":1}
I20260619 04:46:22.101048 12654 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44025
I20260619 04:46:22.101432 12654 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:22.102147 12654 heartbeater.cc:511] Master 127.3.243.190:44025 requested a full tablet report, sending...
I20260619 04:46:22.103708 12331 ts_manager.cc:194] Registered new tserver with Master: 7192b0ba7d3f42e9b9463152e2fb4fca (127.3.243.130:38027)
I20260619 04:46:22.105043 12331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:41301
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:46:22.138867 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44025
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:22.460829 12659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:22.461113 12659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:22.461208 12659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:22.468520 12659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:22.468649 12659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:22.468762 12659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:46:22.478612 12659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:22.479890 12659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:22.481276 12659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:22.494962 12665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:22.494912 12667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:22.495469 12664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:22.496006 12659 server_base.cc:1034] running on GCE node
I20260619 04:46:22.496649 12659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:22.497562 12659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:22.498793 12659 hybrid_clock.cc:648] HybridClock initialized: now 1781844382498747 us; error 40 us; skew 500 ppm
I20260619 04:46:22.499127 12659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:22 dist-test-slave-2qp8 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844382, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:22.502306 12659 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:46:22.504009 12659 webserver.cc:460] Webserver started at http://127.3.243.131:44545/ using document root <none> and password file <none>
I20260619 04:46:22.504745 12659 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:22.504889 12659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:22.505205 12659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:22.508088 12659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "5a791f3a12354f52a237fe8d50ae8117"
format_stamp: "Formatted at 2026-06-19 04:46:22 on dist-test-slave-2qp8"
server_key: "5589c6fe803af78c1efe68faff6bdc4b"
server_key_iv: "a8ad5b7fc9056377da871399025a871a"
server_key_version: "encryptionkey@0"
I20260619 04:46:22.508991 12659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "5a791f3a12354f52a237fe8d50ae8117"
format_stamp: "Formatted at 2026-06-19 04:46:22 on dist-test-slave-2qp8"
server_key: "5589c6fe803af78c1efe68faff6bdc4b"
server_key_iv: "a8ad5b7fc9056377da871399025a871a"
server_key_version: "encryptionkey@0"
I20260619 04:46:22.514366 12659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 04:46:22.517997 12674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:22.519292 12659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:46:22.519464 12659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "5a791f3a12354f52a237fe8d50ae8117"
format_stamp: "Formatted at 2026-06-19 04:46:22 on dist-test-slave-2qp8"
server_key: "5589c6fe803af78c1efe68faff6bdc4b"
server_key_iv: "a8ad5b7fc9056377da871399025a871a"
server_key_version: "encryptionkey@0"
I20260619 04:46:22.519658 12659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:22.542343 12659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:22.546528 12659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:22.546834 12659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:22.547797 12659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:22.549180 12659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:22.549283 12659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:22.549432 12659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:22.549510 12659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:22.582455 12659 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:35319
I20260619 04:46:22.582520 12786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:35319 every 8 connection(s)
I20260619 04:46:22.583745 12659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:46:22.589779 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 12659
I20260619 04:46:22.589958 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:46:22.590478 4046 external_mini_cluster.cc:1428] Setting key 7fa3ecd4aa10dda634d442d0d541f661
Jun 19 04:46:22 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844382, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:46:22.601060 12520 heartbeater.cc:503] Master 127.3.243.190:44025 was elected leader, sending a full tablet report...
I20260619 04:46:22.604360 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:22.586278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:37525 (local address 127.3.243.190:44025)
0619 04:46:22.586498 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:22.586505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:22.587717 (+ 1212us) server_negotiation.cc:407] Connection header received
0619 04:46:22.589657 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:22.589663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:22.589762 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:22.589883 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:22.591612 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.592811 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.594456 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.594875 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.595027 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:22.598569 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:22.598575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:22.598579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:22.598623 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:22.601093 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.601972 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.601979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.601982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.602129 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.602613 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.602616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.602618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.602868 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:22.603040 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:22.603727 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 04:46:22.603936 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 04:46:22.605660 12789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:22.586837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:37525)
0619 04:46:22.587540 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:22.587582 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:22.589366 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:22.590153 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:22.590162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:22.590741 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:22.591438 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:22.591466 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.592963 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.592973 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:22.594307 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:22.594319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.595113 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.595123 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:22.595356 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:22.596226 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:22.596256 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:22.598376 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:22.601251 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:22.601259 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:22.601274 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:22.601862 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:22.602234 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:22.602240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:22.602244 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:22.602505 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:22.603012 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:22.603020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:22.603346 (+ 326us) client_negotiation.cc:769] Sending connection context
0619 04:46:22.603649 (+ 303us) client_negotiation.cc:240] Negotiation successful
0619 04:46:22.604212 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":253,"threads_started":1}
I20260619 04:46:22.606850 12787 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44025
I20260619 04:46:22.607141 12787 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:22.607822 12787 heartbeater.cc:511] Master 127.3.243.190:44025 requested a full tablet report, sending...
I20260619 04:46:22.609191 12331 ts_manager.cc:194] Registered new tserver with Master: 5a791f3a12354f52a237fe8d50ae8117 (127.3.243.131:35319)
I20260619 04:46:22.609941 12331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:37525
I20260619 04:46:22.617982 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab.
Entry for principal oryx/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab.
WARNING: no policy specified for oryx/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab.
Entry for principal oryx/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab.
WARNING: no policy specified for oryx/127.3.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab.
Entry for principal oryx/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab.
I20260619 04:46:22.749135 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 12392
Jun 19 04:46:22 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:46:22.765177 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:22.750251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.3.243.129:33653)
0619 04:46:22.750919 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:22.750929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:22.750963 (+ 34us) server_negotiation.cc:407] Connection header received
0619 04:46:22.751050 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:22.751058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:22.751299 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:22.751491 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:22.752926 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.753939 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.755238 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.755796 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.755949 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:22.759872 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:22.759883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:22.759897 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:22.759935 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:22.762706 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.763319 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.763326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.763329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.763449 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.763743 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.763746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.763748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.764078 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:22.764352 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:22.764593 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 04:46:22.764779 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
I20260619 04:46:22.766479 12515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:22.847503 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 12392
I20260619 04:46:22.863408 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 12526
Jun 19 04:46:22 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:46:22.879164 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:22.864391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.3.243.130:38027)
0619 04:46:22.864926 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:22.864933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:22.864957 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:46:22.865016 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:22.865022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:22.865245 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:22.865416 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:22.866596 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.867586 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.868916 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.869492 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.869626 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:22.873654 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:22.873665 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:22.873679 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:22.873712 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:22.876576 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.877216 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.877223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.877225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.877340 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.877711 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.877715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.877717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.878026 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:22.878344 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:22.878583 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 04:46:22.878834 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":278,"threads_started":1}
I20260619 04:46:22.880421 12649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:22.961474 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 12526
I20260619 04:46:22.977669 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 12659
Jun 19 04:46:22 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844380, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:46:22.994197 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:22.978565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.3.243.131:35319)
0619 04:46:22.979126 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:22.979132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:22.979161 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:46:22.979224 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:22.979230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:22.979446 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:22.979621 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:22.980990 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.981998 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.983260 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:22.983852 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:22.983995 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:22.988478 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:22.988491 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:22.988507 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:22.988550 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:22.991403 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.992042 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.992050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.992052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.992352 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:22.992763 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:22.992767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:22.992770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:22.993196 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:22.993469 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:22.993712 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 04:46:22.993919 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":273,"threads_started":1}
I20260619 04:46:22.995499 12782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:23.078806 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 12659
I20260619 04:46:23.095053 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 12299
I20260619 04:46:23.096422 12361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:23.201177 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 12299
I20260619 04:46:23.221839 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44025
--webserver_interface=127.3.243.190
--webserver_port=40169
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:23.542007 12804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:23.542273 12804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:23.542467 12804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:23.549582 12804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:23.549686 12804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:23.549795 12804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:23.549877 12804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:23.549937 12804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:23.559629 12804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:23.560981 12804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:23.562351 12804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:23.574790 12809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:23.574746 12812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:23.574815 12810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:23.575310 12804 server_base.cc:1034] running on GCE node
I20260619 04:46:23.576392 12804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:23.577848 12804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:23.579084 12804 hybrid_clock.cc:648] HybridClock initialized: now 1781844383579049 us; error 50 us; skew 500 ppm
I20260619 04:46:23.579439 12804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:23 dist-test-slave-2qp8 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844383, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:23.582480 12804 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:46:23.583892 12804 webserver.cc:460] Webserver started at http://127.3.243.190:40169/ using document root <none> and password file <none>
I20260619 04:46:23.584661 12804 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:23.584817 12804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:23.590122 12804 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:46:23.593027 12819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:23.594362 12804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:46:23.594539 12804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "1b15ba73bb98431da6d308479f5896f4"
format_stamp: "Formatted at 2026-06-19 04:46:20 on dist-test-slave-2qp8"
server_key: "520f34686831a634470bc3d31c5855b3"
server_key_iv: "71855425bc1d20535adc1f8d51eb8b3d"
server_key_version: "encryptionkey@0"
I20260619 04:46:23.595480 12804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:23.617631 12804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:23.621668 12804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:23.621938 12804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:23.640558 12804 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44025
I20260619 04:46:23.640576 12870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44025 every 8 connection(s)
I20260619 04:46:23.642062 12804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:46:23.642889 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 12804
I20260619 04:46:23.643839 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:33653
--local_ip_for_outbound_sockets=127.3.243.129
--tserver_master_addrs=127.3.243.190:44025
--webserver_port=45713
--webserver_interface=127.3.243.129
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.243.129
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:46:23.649174 12871 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:46:23.652375 12871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: Bootstrap starting.
I20260619 04:46:23.671034 12871 log.cc:824] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:23.679563 12871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:46:23.680065 12871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: Bootstrap complete.
I20260619 04:46:23.689007 12871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } }
I20260619 04:46:23.689877 12871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b15ba73bb98431da6d308479f5896f4, State: Initialized, Role: FOLLOWER
I20260619 04:46:23.690522 12871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } }
I20260619 04:46:23.690775 12871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:46:23.690922 12871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:46:23.691107 12871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:46:23.694937 12871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } }
I20260619 04:46:23.695441 12871 leader_election.cc:302] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b15ba73bb98431da6d308479f5896f4; no voters:
I20260619 04:46:23.696056 12871 leader_election.cc:288] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 04:46:23.696269 12875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:46:23.697340 12875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [term 2 LEADER]: Becoming Leader. State: Replica: 1b15ba73bb98431da6d308479f5896f4, State: Running, Role: LEADER
I20260619 04:46:23.697956 12875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } }
I20260619 04:46:23.698158 12871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:46:23.700615 12877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b15ba73bb98431da6d308479f5896f4. Latest consensus state: current_term: 2 leader_uuid: "1b15ba73bb98431da6d308479f5896f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } } }
I20260619 04:46:23.700652 12876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1b15ba73bb98431da6d308479f5896f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b15ba73bb98431da6d308479f5896f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44025 } } }
I20260619 04:46:23.700991 12877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:23.701071 12876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:23.708935 12880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:46:23.716262 12880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:46:23.718091 12880 catalog_manager.cc:1259] Loaded cluster ID: fe798862b3a3483a945df11051defbfe
I20260619 04:46:23.718291 12880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:46:23.723500 12880 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:46:23.725459 12880 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b15ba73bb98431da6d308479f5896f4: Loaded TSK: 0
I20260619 04:46:23.727474 12880 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 04:46:24.003197 12873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:24.003525 12873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:24.003696 12873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:24.011325 12873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:24.011473 12873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:24.011659 12873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:46:24.021641 12873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.129.keytab
--principal=oryx/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:33653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=45713
--tserver_master_addrs=127.3.243.190:44025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:24.023030 12873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:24.024518 12873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:24.038908 12898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:24.039505 12900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:24.039875 12897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:24.040751 12873 server_base.cc:1034] running on GCE node
I20260619 04:46:24.041478 12873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:24.042459 12873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:24.043732 12873 hybrid_clock.cc:648] HybridClock initialized: now 1781844384043657 us; error 62 us; skew 500 ppm
I20260619 04:46:24.044091 12873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:24 dist-test-slave-2qp8 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844384, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:24.047591 12873 init.cc:377] Logged in from keytab as oryx/127.3.243.129@KRBTEST.COM (short username oryx)
I20260619 04:46:24.049238 12873 webserver.cc:460] Webserver started at http://127.3.243.129:45713/ using document root <none> and password file <none>
I20260619 04:46:24.049942 12873 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:24.050099 12873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:24.055595 12873 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:46:24.058319 12907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:24.059791 12873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:46:24.059964 12873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "afb832b9146b4b15a12ca79478898d05"
format_stamp: "Formatted at 2026-06-19 04:46:21 on dist-test-slave-2qp8"
server_key: "a96b39c26030b24c362cf88117216122"
server_key_iv: "31cbd90aa4affcb76e2e02e640e3966b"
server_key_version: "encryptionkey@0"
I20260619 04:46:24.060976 12873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:24.085587 12873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:24.089790 12873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:24.090072 12873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:24.091127 12873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:24.092799 12873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:24.092922 12873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:24.093050 12873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:24.093149 12873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:24.127002 12873 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:33653
I20260619 04:46:24.127092 13019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:33653 every 8 connection(s)
I20260619 04:46:24.128558 12873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:46:24.136778 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 12873
I20260619 04:46:24.137404 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:38027
--local_ip_for_outbound_sockets=127.3.243.130
--tserver_master_addrs=127.3.243.190:44025
--webserver_port=43895
--webserver_interface=127.3.243.130
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.243.130
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:46:24 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:24 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 04:46:24.473968 13024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:24.474285 13024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:24.474452 13024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:24.481861 13024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:24.481995 13024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:24.482172 13024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:46:24.491739 13024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.130.keytab
--principal=oryx/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:38027
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=43895
--tserver_master_addrs=127.3.243.190:44025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:24.493098 13024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:24.494388 13024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:24.512928 13030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:24.512974 13031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:24.514357 13033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:24.514585 13024 server_base.cc:1034] running on GCE node
I20260619 04:46:24.515704 13024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:24.517427 13024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:24.518739 13024 hybrid_clock.cc:648] HybridClock initialized: now 1781844384518671 us; error 40 us; skew 500 ppm
I20260619 04:46:24.519299 13024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:24 dist-test-slave-2qp8 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844384, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:24.524994 13024 init.cc:377] Logged in from keytab as oryx/127.3.243.130@KRBTEST.COM (short username oryx)
I20260619 04:46:24.527429 13024 webserver.cc:460] Webserver started at http://127.3.243.130:43895/ using document root <none> and password file <none>
I20260619 04:46:24.528738 13024 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:24.528961 13024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:24.538808 13024 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 04:46:24.543344 13040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:24.545598 13024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 04:46:24.545854 13024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "7192b0ba7d3f42e9b9463152e2fb4fca"
format_stamp: "Formatted at 2026-06-19 04:46:22 on dist-test-slave-2qp8"
server_key: "f15e547802b493c52fdf9c95f70de5b5"
server_key_iv: "ff8a446647ef2b33215f5239f38ba87f"
server_key_version: "encryptionkey@0"
I20260619 04:46:24.547888 13024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:24.602586 13024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:24.610339 13024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:24.610790 13024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:24.612586 13024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:24.614993 13024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:24.615165 13024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:24.615317 13024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:24.615427 13024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:24.659081 13152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:38027 every 8 connection(s)
I20260619 04:46:24.659088 13024 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:38027
I20260619 04:46:24.661674 13024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:46:24.664875 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 13024
I20260619 04:46:24.665783 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:35319
--local_ip_for_outbound_sockets=127.3.243.131
--tserver_master_addrs=127.3.243.190:44025
--webserver_port=44545
--webserver_interface=127.3.243.131
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.243.131
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:46:24 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:24 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
W20260619 04:46:25.003378 13157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:25.003679 13157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:25.003871 13157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:25.011319 13157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:25.011468 13157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:25.011657 13157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:46:25.021905 13157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/krb5kdc/oryx_127.3.243.131.keytab
--principal=oryx/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:35319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=44545
--tserver_master_addrs=127.3.243.190:44025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:25.023237 13157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:25.024704 13157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:25.041975 13164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:25.042129 13163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:25.050002 13166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:25.050738 13157 server_base.cc:1034] running on GCE node
I20260619 04:46:25.051822 13157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:25.053601 13157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:25.055001 13157 hybrid_clock.cc:648] HybridClock initialized: now 1781844385054894 us; error 88 us; skew 500 ppm
I20260619 04:46:25.055545 13157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:25 dist-test-slave-2qp8 krb5kdc[12286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844385, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:25.061661 13157 init.cc:377] Logged in from keytab as oryx/127.3.243.131@KRBTEST.COM (short username oryx)
I20260619 04:46:25.064693 13157 webserver.cc:460] Webserver started at http://127.3.243.131:44545/ using document root <none> and password file <none>
I20260619 04:46:25.066028 13157 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:25.066242 13157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:25.077105 13157 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20260619 04:46:25.082226 13173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:25.084815 13157 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260619 04:46:25.085062 13157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "5a791f3a12354f52a237fe8d50ae8117"
format_stamp: "Formatted at 2026-06-19 04:46:22 on dist-test-slave-2qp8"
server_key: "5589c6fe803af78c1efe68faff6bdc4b"
server_key_iv: "a8ad5b7fc9056377da871399025a871a"
server_key_version: "encryptionkey@0"
I20260619 04:46:25.087096 13157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:25.142015 13157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:25.150146 13157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:25.150743 13157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:25.152683 13157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:25.155228 13157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:25.155380 13157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:25.155609 13157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:25.155717 13157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:25.215469 13157 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:35319
I20260619 04:46:25.215562 13285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:35319 every 8 connection(s)
I20260619 04:46:25.218222 13157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:46:25.220330 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 13157
Jun 19 04:46:25 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:25 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:25.263357 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:25.222393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.3.243.190:44025)
0619 04:46:25.237467 (+ 15074us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:25.237474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:25.238244 (+ 770us) server_negotiation.cc:407] Connection header received
0619 04:46:25.238312 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:25.238317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:25.238403 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:25.238504 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:25.241355 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.243256 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.245978 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.246621 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.246754 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:25.251865 (+ 5111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:25.251878 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:25.251898 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:25.251950 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:25.256428 (+ 4478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:25.258471 (+ 2043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:25.258486 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:25.258496 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:25.258699 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:25.259205 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:25.259213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:25.259217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:25.259920 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:25.260286 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:25.261729 (+ 1443us) server_negotiation.cc:299] Negotiation successful
0619 04:46:25.262219 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14941,"thread_start_us":14824,"threads_started":1}
W20260619 04:46:25.581447 13023 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.439s user 0.011s sys 0.007s
W20260619 04:46:25.583699 13022 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:24.131582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:51575 (local address 127.3.243.190:44025)
0619 04:46:24.132410 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:24.132430 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:24.132662 (+ 232us) server_negotiation.cc:407] Connection header received
0619 04:46:24.134883 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:24.134902 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:24.135238 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:24.135608 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:24.137273 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:24.138553 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:24.140061 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:24.140807 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:24.141017 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:25.583174 (+1442157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:51575: BlockingRecv error: recv got EOF from 127.3.243.129:51575 (error 108)
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":294,"threads_started":1}
I20260619 04:46:25.586287 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:24.131610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:51575)
0619 04:46:24.132479 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:24.132540 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:24.134311 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:24.135747 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:24.135758 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:24.136470 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:24.137065 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:24.137085 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:24.138714 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:24.138720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:24.139897 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:24.139909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:24.141058 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:24.141066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:24.141351 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:24.142050 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:24.142073 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:25.584634 (+1442561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":330,"threads_started":1}
W20260619 04:46:25.587818 13020 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:44025 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.243.190:44025: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Jun 19 04:46:25 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:25 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:25.605193 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:25.588801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:45639)
0619 04:46:25.589066 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:25.589097 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:25.589272 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:25.589770 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:25.589777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:25.590449 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:25.590763 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:25.590780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.592876 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.592884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:25.595163 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:25.595188 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.596978 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.596988 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:25.597261 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:25.598278 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:25.598313 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:25.604731 (+ 6418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":75}
W20260619 04:46:25.605500 13022 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:25.588813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:45639 (local address 127.3.243.190:44025)
0619 04:46:25.589189 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:25.589199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:25.589242 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:46:25.589445 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:25.589453 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:25.589564 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:25.589695 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:25.590985 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.592693 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.595419 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.596403 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.596542 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:25.605108 (+ 8566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:45639: BlockingRecv error: recv got EOF from 127.3.243.129:45639 (error 108)
Metrics: {"server-negotiator.queue_time_us":199}
Jun 19 04:46:25 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:25 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:25.623075 13023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:25.606504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:45543)
0619 04:46:25.607060 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:25.607087 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:25.607296 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:25.608044 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:25.608052 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:25.608692 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:25.608981 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:25.608999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.611273 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.611281 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:25.613594 (+ 2313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:25.613615 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.615048 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.615057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:25.615343 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:25.616411 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:25.616446 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:25.622637 (+ 6191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398}
W20260619 04:46:25.623378 13022 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:25.606638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:45543 (local address 127.3.243.190:44025)
0619 04:46:25.606871 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:25.606882 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:25.607170 (+ 288us) server_negotiation.cc:407] Connection header received
0619 04:46:25.607506 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:25.607515 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:25.607625 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:25.607720 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:25.609228 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.610978 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.613793 (+ 2815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.614825 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.615021 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:25.622942 (+ 7921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:45543: BlockingRecv error: recv got EOF from 127.3.243.129:45543 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 04:46:25.623828 13020 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:46:26.044293 13156 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.352s user 0.014s sys 0.004s
W20260619 04:46:26.045653 13155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:24.664856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:44871 (local address 127.3.243.190:44025)
0619 04:46:24.682497 (+ 17641us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:24.682507 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:24.682539 (+ 32us) server_negotiation.cc:407] Connection header received
0619 04:46:24.682632 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:24.682641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:24.682773 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:24.682963 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:24.685215 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:24.686978 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:24.689745 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:24.690682 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:24.690885 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.045310 (+1354425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:44871: BlockingRecv error: recv got EOF from 127.3.243.130:44871 (error 108)
Metrics: {"server-negotiator.queue_time_us":17431,"thread_start_us":18312,"threads_started":1}
I20260619 04:46:26.046295 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:24.665867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:44871)
0619 04:46:24.678819 (+ 12952us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:24.678885 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:24.681635 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:24.683252 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:24.683267 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:24.684133 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:24.684984 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:24.685013 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:24.687192 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:24.687201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:24.689564 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:24.689582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:24.691007 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:24.691017 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:24.691386 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:24.692391 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:24.692428 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:26.045103 (+1352675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12574,"thread_start_us":12085,"threads_started":1}
W20260619 04:46:26.047533 13153 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:44025 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.243.190:44025: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:26.063462 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:26.048341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:52761)
0619 04:46:26.048637 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:26.048667 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:26.048828 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:26.049179 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:26.049187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:26.049860 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:26.050169 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.050185 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.051221 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.051229 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.053637 (+ 2408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.053659 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.054811 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.054821 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.055090 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.058878 (+ 3788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:26.058895 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:26.063171 (+ 4276us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260619 04:46:26.063586 13155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:26.048558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:52761 (local address 127.3.243.190:44025)
0619 04:46:26.048707 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:26.048712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:26.048727 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:46:26.048962 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:26.048965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:26.049024 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:26.049099 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:26.050340 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.051065 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.053919 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.054562 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.054707 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.063355 (+ 8648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:52761: BlockingRecv error: recv got EOF from 127.3.243.130:52761 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:26.074229 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:26.064625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:45473)
0619 04:46:26.064781 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:26.064801 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:26.064956 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:26.065494 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:26.065498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:26.065804 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:26.065947 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.065954 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.066951 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.066956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.067919 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.067929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.068663 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.068668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.068849 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.069540 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:26.069556 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:26.073869 (+ 4313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 04:46:26.074368 13155 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:26.064940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:45473 (local address 127.3.243.190:44025)
0619 04:46:26.065104 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:26.065108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:26.065121 (+ 13us) server_negotiation.cc:407] Connection header received
0619 04:46:26.065160 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:26.065163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:26.065267 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:26.065351 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:26.066079 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.066780 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.068058 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.068521 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.068651 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.074102 (+ 5451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:45473: BlockingRecv error: recv got EOF from 127.3.243.130:45473 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 04:46:26.074627 13153 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:46:26.368798 13290 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.113s user 0.006s sys 0.009s
W20260619 04:46:26.369869 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:25.232492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:43881 (local address 127.3.243.190:44025)
0619 04:46:25.245219 (+ 12727us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:25.245228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:25.245257 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:46:25.245327 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:25.245335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:25.245441 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:25.245620 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:25.248367 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.250153 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.253239 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.253953 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.254980 (+ 1027us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.369671 (+1114691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:43881: BlockingRecv error: recv got EOF from 127.3.243.131:43881 (error 108)
Metrics: {"server-negotiator.queue_time_us":12523,"thread_start_us":13399,"threads_started":1}
I20260619 04:46:26.370468 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:25.231688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:43881)
0619 04:46:25.236712 (+ 5024us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:25.236759 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:25.239204 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:25.245978 (+ 6774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:25.245991 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:25.246810 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:25.247982 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:25.248020 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.250425 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.250433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:25.253042 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:25.253066 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:25.254270 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:25.254279 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:25.254610 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:25.255334 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:25.255368 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:26.369509 (+1114141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4688,"thread_start_us":4388,"threads_started":1}
W20260619 04:46:26.371333 13286 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:44025 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.243.190:44025: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:26.380594 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:26.372070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:55857)
0619 04:46:26.372282 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:26.372299 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:26.372415 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:26.372694 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:26.372698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:26.373033 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:26.373189 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.373196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.374123 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.374127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.375067 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.375077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.375792 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.375796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.375906 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.376466 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:26.376483 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:26.380309 (+ 3826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260619 04:46:26.380740 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:26.372152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:55857 (local address 127.3.243.190:44025)
0619 04:46:26.372332 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:26.372336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:26.372353 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:46:26.372518 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:26.372522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:26.372579 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:26.372656 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:26.373297 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.374006 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.375193 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.375645 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.375731 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.380521 (+ 4790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:55857: BlockingRecv error: recv got EOF from 127.3.243.131:55857 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:26.389170 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:26.381205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:34225)
0619 04:46:26.381373 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:26.381388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:26.381466 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:26.381849 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:26.381855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:26.382192 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:26.382333 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.382341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.383308 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.383313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.384322 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.384332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.385040 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.385046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.385228 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.385812 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:26.385829 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:26.388889 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 04:46:26.389319 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:26.381311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:34225 (local address 127.3.243.190:44025)
0619 04:46:26.381576 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:26.381581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:26.381599 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:46:26.381646 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:26.381651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:26.381712 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:26.381815 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:26.382475 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.383175 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.384462 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.384886 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.385022 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.389072 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:34225: BlockingRecv error: recv got EOF from 127.3.243.131:34225 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260619 04:46:26.389473 13286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:26 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:26.634231 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:26.624489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:37571)
0619 04:46:26.625087 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:26.625110 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:26.625305 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:26.625736 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:26.625743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:26.626224 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:26.626399 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.626410 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.627493 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.627499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.628588 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:26.628601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.629523 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.629528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:26.629653 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.630336 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:26.630354 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:26.633997 (+ 3643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":347,"threads_started":1}
W20260619 04:46:26.634438 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:26.624581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:37571 (local address 127.3.243.190:44025)
0619 04:46:26.624845 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:26.624855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:26.625224 (+ 369us) server_negotiation.cc:407] Connection header received
0619 04:46:26.625447 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:26.625457 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:26.625563 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:26.625684 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:26.626572 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.627336 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.628746 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:26.629305 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:26.629456 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:26.634210 (+ 4754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:37571: BlockingRecv error: recv got EOF from 127.3.243.129:37571 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Jun 19 04:46:27 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:27 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:27.085479 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:27.075250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:47545)
0619 04:46:27.075838 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:27.075854 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:27.076005 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:27.076379 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:27.076384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:27.076812 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:27.076956 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:27.076964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.077944 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.077953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:27.079157 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:27.079171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.080555 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.080562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:27.080769 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:27.081557 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:27.081605 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:27.085223 (+ 3618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":382,"threads_started":1}
W20260619 04:46:27.085825 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:27.075341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:47545 (local address 127.3.243.190:44025)
0619 04:46:27.075522 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:27.075530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:27.075947 (+ 417us) server_negotiation.cc:407] Connection header received
0619 04:46:27.076169 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:27.076175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:27.076247 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:27.076347 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:27.077094 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.077793 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.079342 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.080005 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.080205 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:27.085544 (+ 5339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:47545: BlockingRecv error: recv got EOF from 127.3.243.130:47545 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 04:46:27 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:27 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:27.399989 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:27.390110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:38709)
0619 04:46:27.390856 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:27.390882 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:27.391002 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:27.391302 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:27.391306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:27.391724 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:27.391864 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:27.391873 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.392881 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.392885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:27.393832 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:27.393843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.394755 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.394760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:27.394904 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:27.395732 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:27.395794 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:27.399755 (+ 3961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":496,"threads_started":1}
W20260619 04:46:27.400214 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:27.390232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:38709 (local address 127.3.243.190:44025)
0619 04:46:27.390399 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:27.390404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:27.390964 (+ 560us) server_negotiation.cc:407] Connection header received
0619 04:46:27.391103 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:27.391106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:27.391178 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:27.391261 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:27.391989 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.392748 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.394016 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.394500 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.394593 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:27.399933 (+ 5340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:38709: BlockingRecv error: recv got EOF from 127.3.243.131:38709 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 04:46:27 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:27 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:27.644482 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:27.635170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:38151)
0619 04:46:27.635609 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:27.635629 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:27.635756 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:27.636074 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:27.636077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:27.636545 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:27.636721 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:27.636730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.637766 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.637771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:27.638864 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:27.638877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.639481 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.639486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:27.639596 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:27.640225 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:27.640242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:27.644153 (+ 3911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":248,"threads_started":1}
W20260619 04:46:27.644752 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:27.635239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:38151 (local address 127.3.243.190:44025)
0619 04:46:27.635460 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:27.635466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:27.635712 (+ 246us) server_negotiation.cc:407] Connection header received
0619 04:46:27.635861 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:27.635864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:27.635934 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:27.636021 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:27.636847 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.637579 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.638981 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:27.639355 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:27.639438 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:27.644480 (+ 5042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:38151: BlockingRecv error: recv got EOF from 127.3.243.129:38151 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 04:46:28 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:28 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:28.095567 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:28.086448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:60353)
0619 04:46:28.086993 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:28.087009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:28.087164 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:28.087505 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:28.087510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:28.087952 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:28.088186 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:28.088201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.089226 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.089231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:28.090152 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:28.090163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.090879 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.090885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:28.091097 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:28.091672 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:28.091690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:28.095197 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":302,"threads_started":1}
W20260619 04:46:28.095583 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:28.086556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:60353 (local address 127.3.243.190:44025)
0619 04:46:28.086819 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:28.086826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:28.087092 (+ 266us) server_negotiation.cc:407] Connection header received
0619 04:46:28.087288 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:28.087294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:28.087368 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:28.087477 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:28.088329 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.089094 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.090308 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.090728 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.090861 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:28.095377 (+ 4516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:60353: BlockingRecv error: recv got EOF from 127.3.243.130:60353 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 04:46:28 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:28 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:28.410146 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:28.401058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:60125)
0619 04:46:28.401546 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:28.401563 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:28.401684 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:28.401973 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:28.401976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:28.402383 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:28.402560 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:28.402568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.403661 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.403665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:28.404742 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:28.404754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.405593 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.405600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:28.405772 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:28.406386 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:28.406405 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:28.409816 (+ 3411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":283,"threads_started":1}
W20260619 04:46:28.410280 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:28.401162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:60125 (local address 127.3.243.190:44025)
0619 04:46:28.401438 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:28.401445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:28.401641 (+ 196us) server_negotiation.cc:407] Connection header received
0619 04:46:28.401780 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:28.401784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:28.401845 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:28.401927 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:28.402684 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.403526 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.404886 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.405438 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.405607 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:28.410016 (+ 4409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:60125: BlockingRecv error: recv got EOF from 127.3.243.131:60125 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Jun 19 04:46:28 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:28 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:28.654378 13299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:28.645607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:50881)
0619 04:46:28.646350 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:28.646374 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:28.646504 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:28.646805 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:28.646810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:28.647283 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:28.647482 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:28.647495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.648534 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.648538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:28.649469 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:28.649480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.650133 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.650137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:28.650264 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:28.650824 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:28.650841 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:28.654150 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":519,"threads_started":1}
W20260619 04:46:28.654608 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:28.645836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:50881 (local address 127.3.243.190:44025)
0619 04:46:28.646014 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:28.646021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:28.646469 (+ 448us) server_negotiation.cc:407] Connection header received
0619 04:46:28.646606 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:28.646610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:28.646671 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:28.646748 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:28.647613 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.648396 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.649589 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:28.650012 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:28.650098 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:28.654371 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:50881: BlockingRecv error: recv got EOF from 127.3.243.129:50881 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 04:46:29 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:29 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:29.106153 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:29.096658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:46973)
0619 04:46:29.097181 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:29.097198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:29.097312 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:29.097612 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:29.097615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:29.097974 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:29.098191 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:29.098202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.099482 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.099487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:29.100765 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:29.100778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.101622 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.101627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:29.101751 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:29.102376 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:29.102392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:29.105802 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
W20260619 04:46:29.106312 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:29.096709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:46973 (local address 127.3.243.190:44025)
0619 04:46:29.096877 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:29.096881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:29.097270 (+ 389us) server_negotiation.cc:407] Connection header received
0619 04:46:29.097426 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:29.097430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:29.097492 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:29.097565 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:29.098311 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.099242 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.100938 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.101503 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.101593 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:29.106033 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:46973: BlockingRecv error: recv got EOF from 127.3.243.130:46973 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 04:46:29 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:29 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:29.420025 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:29.411138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:48611)
0619 04:46:29.411741 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:29.411760 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:29.412038 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:29.412459 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:29.412463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:29.412859 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:29.413023 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:29.413040 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.414033 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.414042 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:29.414963 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:29.414973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.415666 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.415671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:29.415869 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:29.416552 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:29.416570 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:29.419681 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":340,"threads_started":1}
W20260619 04:46:29.420074 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:29.411230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:48611 (local address 127.3.243.190:44025)
0619 04:46:29.411473 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:29.411481 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:29.411884 (+ 403us) server_negotiation.cc:407] Connection header received
0619 04:46:29.412230 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:29.412236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:29.412323 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:29.412428 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:29.413175 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.413891 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.415102 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.415531 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.415658 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:29.419879 (+ 4221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:48611: BlockingRecv error: recv got EOF from 127.3.243.131:48611 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 19 04:46:29 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:29 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:29.665372 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:29.655495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:48017)
0619 04:46:29.655987 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:29.656008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:29.656174 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:29.656599 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:29.656603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:29.656983 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:29.657144 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:29.657153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.658259 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.658263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:29.659270 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:29.659283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.660256 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.660260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:29.660398 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:29.661073 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:29.661091 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:29.664916 (+ 3825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":274,"threads_started":1}
W20260619 04:46:29.665442 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:29.655525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:48017 (local address 127.3.243.190:44025)
0619 04:46:29.655718 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:29.655722 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:29.656085 (+ 363us) server_negotiation.cc:407] Connection header received
0619 04:46:29.656292 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:29.656296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:29.656414 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:29.656548 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:29.657347 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.658128 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.659413 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:29.660065 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:29.660221 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:29.665128 (+ 4907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:48017: BlockingRecv error: recv got EOF from 127.3.243.129:48017 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 19 04:46:30 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:30 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:30.115864 13303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:30.107201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:58551)
0619 04:46:30.107658 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:30.107674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:30.107828 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:30.108183 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:30.108187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:30.108594 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:30.108745 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:30.108753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.109744 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.109752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:30.110839 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:30.110852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.111633 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.111639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:30.111801 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:30.112390 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:30.112406 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:30.115527 (+ 3121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":242,"threads_started":1}
W20260619 04:46:30.115952 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:30.107222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:58551 (local address 127.3.243.190:44025)
0619 04:46:30.107415 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:30.107422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:30.107754 (+ 332us) server_negotiation.cc:407] Connection header received
0619 04:46:30.107961 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:30.107966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:30.108026 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:30.108145 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:30.108888 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.109624 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.110976 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.111487 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.111612 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:30.115714 (+ 4102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:58551: BlockingRecv error: recv got EOF from 127.3.243.130:58551 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 04:46:30 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:30 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:30.429726 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:30.421010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:39947)
0619 04:46:30.421558 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:30.421573 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:30.421690 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:30.421985 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:30.421988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:30.422346 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:30.422515 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:30.422524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.423496 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.423500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:30.424603 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:30.424615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.425211 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.425215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:30.425330 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:30.425972 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:30.425990 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:30.429484 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":310,"threads_started":1}
W20260619 04:46:30.429929 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:30.421039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:39947 (local address 127.3.243.190:44025)
0619 04:46:30.421225 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:30.421229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:30.421657 (+ 428us) server_negotiation.cc:407] Connection header received
0619 04:46:30.421786 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:30.421789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:30.421851 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:30.421930 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:30.422633 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.423373 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.424721 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.425097 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.425184 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:30.429678 (+ 4494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:39947: BlockingRecv error: recv got EOF from 127.3.243.131:39947 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 04:46:30 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:30 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:30.674525 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:30.666242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:39153)
0619 04:46:30.666695 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:30.666714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:30.666825 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:30.667133 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:30.667136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:30.667586 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:30.667766 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:30.667775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.668708 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.668712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:30.669627 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:30.669638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.670237 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.670241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:30.670374 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:30.670915 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:30.670931 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:30.674208 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":263,"threads_started":1}
W20260619 04:46:30.674690 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:30.666337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:39153 (local address 127.3.243.190:44025)
0619 04:46:30.666600 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:30.666607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:30.666791 (+ 184us) server_negotiation.cc:407] Connection header received
0619 04:46:30.666944 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:30.666948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:30.667013 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:30.667094 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:30.667880 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.668596 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.669738 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:30.670114 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:30.670199 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:30.674399 (+ 4200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:39153: BlockingRecv error: recv got EOF from 127.3.243.129:39153 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 19 04:46:31 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:31 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:31.130172 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:31.117034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:40521)
0619 04:46:31.117569 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:31.117585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:31.117692 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:31.117963 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:31.117966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:31.118399 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:31.118570 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:31.118579 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.119538 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.119541 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:31.120481 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:31.120492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.121133 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.121137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:31.121259 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:31.126011 (+ 4752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:31.126028 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:31.129853 (+ 3825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":298,"threads_started":1}
W20260619 04:46:31.130327 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:31.117192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:40521 (local address 127.3.243.190:44025)
0619 04:46:31.117345 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:31.117349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:31.117653 (+ 304us) server_negotiation.cc:407] Connection header received
0619 04:46:31.117781 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:31.117784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:31.117842 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:31.117919 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:31.118679 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.119405 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.120597 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.121016 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.121108 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:31.130031 (+ 8923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:40521: BlockingRecv error: recv got EOF from 127.3.243.130:40521 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 19 04:46:31 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:31 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:31.439853 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:31.430775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:43931)
0619 04:46:31.431337 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:31.431355 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:31.431480 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:31.431804 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:31.431810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:31.432268 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:31.432421 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:31.432430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.433398 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.433402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:31.434282 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:31.434292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.435097 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.435104 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:31.435311 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:31.436038 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:31.436058 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:31.439595 (+ 3537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":299,"threads_started":1}
W20260619 04:46:31.439988 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:31.430814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:43931 (local address 127.3.243.190:44025)
0619 04:46:31.430993 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:31.430998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:31.431441 (+ 443us) server_negotiation.cc:407] Connection header received
0619 04:46:31.431580 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:31.431584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:31.431642 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:31.431722 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:31.432550 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.433268 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.434413 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.434931 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.435031 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:31.439772 (+ 4741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:43931: BlockingRecv error: recv got EOF from 127.3.243.131:43931 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 04:46:31 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:31 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:31.684180 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:31.675654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:37855)
0619 04:46:31.676178 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:31.676198 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:31.676354 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:31.676681 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:31.676685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:31.677106 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:31.677260 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:31.677268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.678282 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.678288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:31.679178 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:31.679189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.679877 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.679882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:31.680079 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:31.680651 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:31.680667 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:31.683790 (+ 3123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":269,"threads_started":1}
W20260619 04:46:31.684314 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:31.675744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:37855 (local address 127.3.243.190:44025)
0619 04:46:31.676022 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:31.676028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:31.676282 (+ 254us) server_negotiation.cc:407] Connection header received
0619 04:46:31.676481 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:31.676493 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:31.676552 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:31.676651 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:31.677442 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.678147 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.679313 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:31.679732 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:31.679865 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:31.683992 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:37855: BlockingRecv error: recv got EOF from 127.3.243.129:37855 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 19 04:46:32 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:32 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:32.140265 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:32.131221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:49119)
0619 04:46:32.131686 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:32.131701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:32.131854 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:32.132207 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:32.132211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:32.132610 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:32.132782 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:32.132791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.133755 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.133761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:32.134677 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:32.134687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.135430 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.135436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:32.135604 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:32.136214 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:32.136232 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:32.139908 (+ 3676us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":256,"threads_started":1}
W20260619 04:46:32.140307 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:32.131292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:49119 (local address 127.3.243.190:44025)
0619 04:46:32.131517 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:32.131525 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:32.131780 (+ 255us) server_negotiation.cc:407] Connection header received
0619 04:46:32.131989 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:32.131993 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:32.132051 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:32.132180 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:32.132925 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.133616 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.134813 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.135240 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.135387 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:32.140079 (+ 4692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:49119: BlockingRecv error: recv got EOF from 127.3.243.130:49119 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 04:46:32 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:32 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:32.450181 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:32.440936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:45009)
0619 04:46:32.441387 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:32.441403 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:32.441545 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:32.441839 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:32.441842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:32.442221 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:32.442384 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:32.442392 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.443497 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.443502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:32.444482 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:32.444493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.445198 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.445202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:32.445326 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:32.445981 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:32.446000 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:32.449956 (+ 3956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":258,"threads_started":1}
W20260619 04:46:32.450392 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:32.440977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:45009 (local address 127.3.243.190:44025)
0619 04:46:32.441208 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:32.441219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:32.441514 (+ 295us) server_negotiation.cc:407] Connection header received
0619 04:46:32.441645 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:32.441649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:32.441712 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:32.441813 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:32.442537 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.443352 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.444635 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.445055 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.445158 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:32.450166 (+ 5008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:45009: BlockingRecv error: recv got EOF from 127.3.243.131:45009 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 19 04:46:32 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:32 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:32.694319 13311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:32.685194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:59385)
0619 04:46:32.685851 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:32.685872 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:32.686029 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:32.686348 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:32.686353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:32.686811 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:32.686994 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:32.687004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.687991 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.687998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:32.689042 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:32.689053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.689802 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.689808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:32.689992 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:32.690553 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:32.690569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:32.694063 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":288,"threads_started":1}
W20260619 04:46:32.694555 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:32.685361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:59385 (local address 127.3.243.190:44025)
0619 04:46:32.685586 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:32.685592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:32.685956 (+ 364us) server_negotiation.cc:407] Connection header received
0619 04:46:32.686139 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:32.686142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:32.686200 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:32.686310 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:32.687143 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.687852 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.689206 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:32.689647 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:32.689775 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:32.694332 (+ 4557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:59385: BlockingRecv error: recv got EOF from 127.3.243.129:59385 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 04:46:33 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:33 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:33.150076 13312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:33.141200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:50055)
0619 04:46:33.141728 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:33.141744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:33.141851 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:33.142124 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:33.142127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:33.142549 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:33.142721 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:33.142729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.143685 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.143689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:33.144674 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:33.144685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.145380 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.145384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:33.145495 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:33.146055 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:33.146071 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:33.149812 (+ 3741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":275,"threads_started":1}
W20260619 04:46:33.150167 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:33.141270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:50055 (local address 127.3.243.190:44025)
0619 04:46:33.141504 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:33.141513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:33.141823 (+ 310us) server_negotiation.cc:407] Connection header received
0619 04:46:33.141948 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:33.141951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:33.142009 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:33.142088 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:33.142835 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.143576 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.144788 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.145221 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.145309 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:33.149943 (+ 4634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:50055: BlockingRecv error: recv got EOF from 127.3.243.130:50055 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 04:46:33 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:33 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:33.461102 13313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:33.451606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:53857)
0619 04:46:33.452300 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:33.452318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:33.452487 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:33.452823 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:33.452827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:33.453237 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:33.453419 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:33.453428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.454452 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.454457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:33.455423 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:33.455436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.456556 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.456561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:33.456741 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:33.457413 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:33.457430 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:33.460768 (+ 3338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":324,"threads_started":1}
W20260619 04:46:33.461122 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:33.451664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:53857 (local address 127.3.243.190:44025)
0619 04:46:33.451939 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:33.451947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:33.452390 (+ 443us) server_negotiation.cc:407] Connection header received
0619 04:46:33.452611 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:33.452615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:33.452675 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:33.452758 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:33.453548 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.454266 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.455572 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.456196 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.456376 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:33.460932 (+ 4556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:53857: BlockingRecv error: recv got EOF from 127.3.243.131:53857 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 04:46:33 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:33 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:33.704877 13314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:33.695501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:58875)
0619 04:46:33.696068 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:33.696087 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:33.696280 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:33.696644 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:33.696649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:33.697120 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:33.697385 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:33.697412 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.698792 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.698796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:33.699913 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:33.699925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.700679 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.700697 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:33.700814 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:33.701420 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:33.701436 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:33.704635 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":303,"threads_started":1}
W20260619 04:46:33.705078 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:33.695703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:58875 (local address 127.3.243.190:44025)
0619 04:46:33.695956 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:33.695963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:33.696193 (+ 230us) server_negotiation.cc:407] Connection header received
0619 04:46:33.696419 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:33.696425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:33.696501 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:33.696611 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:33.697561 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.698672 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.700028 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:33.700530 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:33.700625 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:33.704815 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:58875: BlockingRecv error: recv got EOF from 127.3.243.129:58875 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 19 04:46:34 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:34 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:34.160244 13315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:34.151133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:57231)
0619 04:46:34.151692 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:34.151709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:34.151874 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:34.152386 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:34.152391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:34.152827 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:34.153004 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:34.153013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.154107 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.154110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:34.155068 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:34.155079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.155849 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.155854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:34.155972 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:34.156735 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:34.156753 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:34.159931 (+ 3178us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":300,"threads_started":1}
W20260619 04:46:34.160387 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:34.151210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:57231 (local address 127.3.243.190:44025)
0619 04:46:34.151382 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:34.151386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:34.151804 (+ 418us) server_negotiation.cc:407] Connection header received
0619 04:46:34.152015 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:34.152020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:34.152175 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:34.152272 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:34.153129 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.153981 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.155210 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.155681 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.155795 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:34.160155 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:57231: BlockingRecv error: recv got EOF from 127.3.243.130:57231 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 19 04:46:34 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:34 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:34.471767 13316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:34.462437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:48641)
0619 04:46:34.462984 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:34.463000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:34.463123 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:34.463491 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:34.463496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:34.463959 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:34.464229 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:34.464240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.465512 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.465517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:34.466456 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:34.466468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.467231 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.467237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:34.467405 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:34.468050 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:34.468067 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:34.471426 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":296,"threads_started":1}
W20260619 04:46:34.471846 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:34.462491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:48641 (local address 127.3.243.190:44025)
0619 04:46:34.462670 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:34.462674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:34.463093 (+ 419us) server_negotiation.cc:407] Connection header received
0619 04:46:34.463247 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:34.463251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:34.463314 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:34.463437 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:34.464408 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.465364 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.466600 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.467051 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.467192 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:34.471609 (+ 4417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:48641: BlockingRecv error: recv got EOF from 127.3.243.131:48641 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 04:46:34 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:34 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:34.715842 13317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:34.705881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:39021)
0619 04:46:34.706402 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:34.706422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:34.706529 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:34.706855 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:34.706858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:34.707248 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:34.707436 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:34.707445 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.708538 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.708543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:34.709577 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:34.709592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.710368 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.710376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:34.710578 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:34.711207 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:34.711230 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:34.715501 (+ 4271us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":231,"threads_started":1}
W20260619 04:46:34.715940 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:34.706017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:39021 (local address 127.3.243.190:44025)
0619 04:46:34.706254 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:34.706260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:34.706495 (+ 235us) server_negotiation.cc:407] Connection header received
0619 04:46:34.706625 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:34.706628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:34.706698 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:34.706793 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:34.707561 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.708385 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.709701 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:34.710107 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:34.710221 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:34.715692 (+ 5471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:39021: BlockingRecv error: recv got EOF from 127.3.243.129:39021 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 04:46:35 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:35 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:35.170814 13318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:35.161343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:43671)
0619 04:46:35.161922 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:35.161939 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:35.162055 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:35.162361 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:35.162364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:35.162795 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:35.162945 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:35.162955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.163979 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.163984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:35.165384 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:35.165397 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.166318 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.166323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:35.166451 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:35.166990 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:35.167008 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:35.170475 (+ 3467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":288,"threads_started":1}
W20260619 04:46:35.170897 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:35.161348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:43671 (local address 127.3.243.190:44025)
0619 04:46:35.161521 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:35.161525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:35.162013 (+ 488us) server_negotiation.cc:407] Connection header received
0619 04:46:35.162151 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:35.162154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:35.162210 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:35.162290 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:35.163064 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.163821 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.165562 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.166157 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.166249 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:35.170648 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:43671: BlockingRecv error: recv got EOF from 127.3.243.130:43671 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 04:46:35 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:35 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:35.482054 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:35.472792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:43297)
0619 04:46:35.473316 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:35.473332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:35.473468 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:35.473745 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:35.473748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:35.474165 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:35.474302 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:35.474310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.475547 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.475553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:35.476584 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:35.476596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.477322 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.477328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:35.477504 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:35.478113 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:35.478130 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:35.481830 (+ 3700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":301,"threads_started":1}
W20260619 04:46:35.482285 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:35.472921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:43297 (local address 127.3.243.190:44025)
0619 04:46:35.473146 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:35.473150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:35.473435 (+ 285us) server_negotiation.cc:407] Connection header received
0619 04:46:35.473561 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:35.473564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:35.473620 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:35.473701 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:35.474487 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.475408 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.476730 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.477168 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.477300 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:35.482054 (+ 4754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:43297: BlockingRecv error: recv got EOF from 127.3.243.131:43297 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 04:46:35 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:35 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:35.725656 13320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:35.716814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:59149)
0619 04:46:35.717302 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:35.717323 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:35.717444 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:35.717720 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:35.717723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:35.718097 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:35.718241 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:35.718250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.719180 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.719183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:35.720136 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:35.720148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.720885 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.720889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:35.721011 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:35.721638 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:35.721655 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:35.725310 (+ 3655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":284,"threads_started":1}
W20260619 04:46:35.725812 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:35.716865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:59149 (local address 127.3.243.190:44025)
0619 04:46:35.717089 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:35.717098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:35.717402 (+ 304us) server_negotiation.cc:407] Connection header received
0619 04:46:35.717539 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:35.717542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:35.717601 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:35.717680 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:35.718375 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.719073 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.720257 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:35.720768 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:35.720864 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:35.725517 (+ 4653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:59149: BlockingRecv error: recv got EOF from 127.3.243.129:59149 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 04:46:36 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:36 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:36.185024 13321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:36.171804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:45805)
0619 04:46:36.172363 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:36.172378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:36.172496 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:36.172765 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:36.172768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:36.173138 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:36.173281 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:36.173289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.174202 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.174205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:36.175148 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:36.175159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.175777 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.175781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:36.175891 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:36.180650 (+ 4759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:36.180670 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:36.184588 (+ 3918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":343,"threads_started":1}
W20260619 04:46:36.185168 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:36.171919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:45805 (local address 127.3.243.190:44025)
0619 04:46:36.172089 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:36.172093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:36.172457 (+ 364us) server_negotiation.cc:407] Connection header received
0619 04:46:36.172577 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:36.172580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:36.172641 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:36.172720 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:36.173399 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.174084 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.175268 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.175654 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.175740 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:36.184861 (+ 9121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:45805: BlockingRecv error: recv got EOF from 127.3.243.130:45805 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 04:46:36 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:36 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:36.491731 13322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:36.483031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:57499)
0619 04:46:36.483558 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:36.483574 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:36.483687 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:36.483989 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:36.483992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:36.484473 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:36.484626 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:36.484634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.485628 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.485631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:36.486583 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:36.486593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.487454 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.487459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:36.487653 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:36.488363 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:36.488383 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:36.491510 (+ 3127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":311,"threads_started":1}
W20260619 04:46:36.491918 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:36.483139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:57499 (local address 127.3.243.190:44025)
0619 04:46:36.483424 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:36.483430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:36.483653 (+ 223us) server_negotiation.cc:407] Connection header received
0619 04:46:36.483784 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:36.483787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:36.483863 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:36.483945 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:36.484751 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.485507 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.486718 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.487265 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.487410 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:36.491678 (+ 4268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:57499: BlockingRecv error: recv got EOF from 127.3.243.131:57499 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Jun 19 04:46:36 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:36 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:36.735386 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:36.726798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:58319)
0619 04:46:36.727282 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:36.727303 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:36.727444 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:36.727720 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:36.727723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:36.728146 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:36.728390 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:36.728406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.729307 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.729311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:36.730198 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:36.730209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.730842 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.730846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:36.730968 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:36.731607 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:36.731624 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:36.735139 (+ 3515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":290,"threads_started":1}
W20260619 04:46:36.735777 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:36.726914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:58319 (local address 127.3.243.190:44025)
0619 04:46:36.727169 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:36.727175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:36.727430 (+ 255us) server_negotiation.cc:407] Connection header received
0619 04:46:36.727540 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:36.727543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:36.727602 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:36.727680 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:36.728533 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.729196 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.730314 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:36.730722 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:36.730809 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:36.735548 (+ 4739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:58319: BlockingRecv error: recv got EOF from 127.3.243.129:58319 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 04:46:37 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:37 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:37.195247 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:37.186053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:34511)
0619 04:46:37.186707 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:37.186722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:37.186882 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:37.187260 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:37.187264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:37.187687 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:37.187865 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:37.187874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.189012 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.189020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:37.189920 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:37.189930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.190627 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.190633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:37.190810 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:37.191396 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:37.191431 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:37.194925 (+ 3494us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":413,"threads_started":1}
W20260619 04:46:37.195364 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:37.186181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:34511 (local address 127.3.243.190:44025)
0619 04:46:37.186514 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:37.186521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:37.186810 (+ 289us) server_negotiation.cc:407] Connection header received
0619 04:46:37.187040 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:37.187047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:37.187117 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:37.187229 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:37.188011 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.188874 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.190063 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.190475 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.190599 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:37.195115 (+ 4516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:34511: BlockingRecv error: recv got EOF from 127.3.243.130:34511 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
Jun 19 04:46:37 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:37 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:37.501950 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:37.492932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:58849)
0619 04:46:37.493433 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:37.493449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:37.493560 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:37.493830 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:37.493833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:37.494208 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:37.494518 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:37.494529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.495530 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.495534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:37.496620 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:37.496632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.497570 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.497576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:37.497751 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:37.498371 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:37.498388 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:37.501610 (+ 3222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":281,"threads_started":1}
W20260619 04:46:37.502007 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:37.492971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:58849 (local address 127.3.243.190:44025)
0619 04:46:37.493136 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:37.493140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:37.493519 (+ 379us) server_negotiation.cc:407] Connection header received
0619 04:46:37.493649 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:37.493653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:37.493708 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:37.493788 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:37.494630 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.495383 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.496722 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.497435 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.497557 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:37.501784 (+ 4227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:58849: BlockingRecv error: recv got EOF from 127.3.243.131:58849 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 04:46:37 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:37 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:37.745393 13326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:37.736420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:34263)
0619 04:46:37.736970 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:37.736992 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:37.737110 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:37.737452 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:37.737456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:37.737892 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:37.738071 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:37.738080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.739107 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.739111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:37.740070 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:37.740083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.741060 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.741064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:37.741202 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:37.741852 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:37.741870 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:37.745119 (+ 3249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":320,"threads_started":1}
W20260619 04:46:37.745668 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:37.736469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:34263 (local address 127.3.243.190:44025)
0619 04:46:37.736653 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:37.736657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:37.737073 (+ 416us) server_negotiation.cc:407] Connection header received
0619 04:46:37.737209 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:37.737212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:37.737268 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:37.737391 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:37.738198 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.738961 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.740266 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:37.740925 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:37.741031 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:37.745379 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:34263: BlockingRecv error: recv got EOF from 127.3.243.129:34263 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 04:46:38 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:38 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:38.205518 13327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:38.196195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:36787)
0619 04:46:38.196668 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:38.196684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:38.196821 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:38.197146 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:38.197151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:38.197569 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:38.197766 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:38.197777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.198959 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.198966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:38.200415 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:38.200428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.201128 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.201132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:38.201316 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:38.201927 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:38.201943 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:38.205280 (+ 3337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":275,"threads_started":1}
W20260619 04:46:38.205679 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:38.196235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:36787 (local address 127.3.243.190:44025)
0619 04:46:38.196451 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:38.196460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:38.196760 (+ 300us) server_negotiation.cc:407] Connection header received
0619 04:46:38.196918 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:38.196921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:38.196983 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:38.197064 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:38.197915 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.198777 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.200556 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.201005 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.201097 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:38.205461 (+ 4364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:36787: BlockingRecv error: recv got EOF from 127.3.243.130:36787 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 04:46:38 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:38 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:38.512089 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:38.502954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:33867)
0619 04:46:38.503478 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:38.503493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:38.503671 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:38.503994 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:38.503998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:38.504531 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:38.504693 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:38.504701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.505728 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.505734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:38.506771 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:38.506781 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.507554 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.507560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:38.507740 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:38.508391 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:38.508410 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:38.511749 (+ 3339us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":284,"threads_started":1}
W20260619 04:46:38.512238 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:38.502985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:33867 (local address 127.3.243.190:44025)
0619 04:46:38.503209 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:38.503217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:38.503605 (+ 388us) server_negotiation.cc:407] Connection header received
0619 04:46:38.503788 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:38.503793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:38.503865 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:38.503966 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:38.504833 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.505592 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.506923 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.507366 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.507508 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:38.511952 (+ 4444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:33867: BlockingRecv error: recv got EOF from 127.3.243.131:33867 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 04:46:38 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:38 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:38.755532 13329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:38.746614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:60971)
0619 04:46:38.747148 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:38.747176 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:38.747339 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:38.747679 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:38.747682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:38.748197 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:38.748391 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:38.748401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.749489 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.749493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:38.750430 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:38.750441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.751070 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.751074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:38.751205 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:38.751892 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:38.751909 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:38.755303 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":347,"threads_started":1}
W20260619 04:46:38.755913 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:38.746735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:60971 (local address 127.3.243.190:44025)
0619 04:46:38.746966 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:38.746974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:38.747247 (+ 273us) server_negotiation.cc:407] Connection header received
0619 04:46:38.747433 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:38.747438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:38.747515 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:38.747616 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:38.748518 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.749276 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.750555 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:38.750951 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:38.751040 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:38.755558 (+ 4518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:60971: BlockingRecv error: recv got EOF from 127.3.243.129:60971 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 04:46:39 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:39 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:39.215277 13330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:39.206473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:48171)
0619 04:46:39.206953 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:39.206969 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:39.207086 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:39.207378 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:39.207381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:39.207751 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:39.207893 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:39.207901 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.208895 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.208899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:39.209761 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:39.209772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.210560 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.210565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:39.210728 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:39.211518 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:39.211536 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:39.215027 (+ 3491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":270,"threads_started":1}
W20260619 04:46:39.215447 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:39.206585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:48171 (local address 127.3.243.190:44025)
0619 04:46:39.206850 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:39.206857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:39.207049 (+ 192us) server_negotiation.cc:407] Connection header received
0619 04:46:39.207178 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:39.207181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:39.207242 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:39.207321 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:39.208047 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.208771 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.209870 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.210382 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.210545 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:39.215231 (+ 4686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:48171: BlockingRecv error: recv got EOF from 127.3.243.130:48171 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 19 04:46:39 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:39 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.131@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:39.521888 13331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:39.513040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.131:33051)
0619 04:46:39.513482 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:39.513497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:39.513684 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:39.514009 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:39.514014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:39.514421 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:39.514555 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:39.514562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.515567 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.515575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:39.516518 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:39.516529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.517232 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.517238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:39.517413 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:39.518033 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:39.518050 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:39.521574 (+ 3524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":226,"threads_started":1}
W20260619 04:46:39.521982 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:39.513110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:33051 (local address 127.3.243.190:44025)
0619 04:46:39.513326 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:39.513333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:39.513612 (+ 279us) server_negotiation.cc:407] Connection header received
0619 04:46:39.513801 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:39.513812 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:39.513869 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:39.513966 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:39.514692 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.515434 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.516658 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.517081 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.517211 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:39.521760 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.131:33051: BlockingRecv error: recv got EOF from 127.3.243.131:33051 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 04:46:39 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 19 04:46:39 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.129@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:39.765745 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:39.756848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.129:57813)
0619 04:46:39.757305 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:39.757325 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:39.757435 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:39.757757 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:39.757760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:39.758208 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:39.758426 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:39.758440 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.759625 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.759630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:39.760622 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:39.760634 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.761350 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.761356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:39.761538 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:39.762124 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:39.762139 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:39.765446 (+ 3307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
W20260619 04:46:39.766116 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:39.756954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:57813 (local address 127.3.243.190:44025)
0619 04:46:39.757281 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:39.757292 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:39.757401 (+ 109us) server_negotiation.cc:407] Connection header received
0619 04:46:39.757561 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:39.757565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:39.757625 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:39.757719 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:39.758576 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.759485 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.760764 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:39.761197 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:39.761331 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:39.765814 (+ 4483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.129:57813: BlockingRecv error: recv got EOF from 127.3.243.129:57813 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Jun 19 04:46:40 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:40.223917 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 12873
Jun 19 04:46:40 dist-test-slave-2qp8 krb5kdc[12286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.243.130@KRBTEST.COM for oryx/127.3.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260619 04:46:40.225812 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:40.216587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44025 (local address 127.3.243.130:34363)
0619 04:46:40.217081 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:40.217099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:40.217205 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:40.217559 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:40.217562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:40.217903 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:40.218096 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:40.218105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.219572 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.219576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:40.220556 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:40.220569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.221282 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.221286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:40.221402 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:40.222013 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:40.222030 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:40.225587 (+ 3557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.243.190:44025: Server oryx/127.3.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":252,"threads_started":1}
W20260619 04:46:40.226349 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:40.216678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:34363 (local address 127.3.243.190:44025)
0619 04:46:40.216904 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:40.216912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:40.217185 (+ 273us) server_negotiation.cc:407] Connection header received
0619 04:46:40.217324 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:40.217329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:40.217412 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:40.217536 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:40.218254 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.219424 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.220698 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.221142 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.221265 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:40.226137 (+ 4872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.243.130:34363: BlockingRecv error: recv got EOF from 127.3.243.130:34363 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 04:46:40.236438 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:40.225236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.3.243.129:33653)
0619 04:46:40.225956 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:40.225966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:40.226002 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:46:40.226080 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:40.226092 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:40.226446 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:40.226642 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:40.228004 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.229140 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.230243 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.230772 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.230872 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:40.233206 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:40.233214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:40.233227 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:40.233264 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:40.235835 (+ 2571us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 04:46:40.236154 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40364: Request ticket server kudu/127.3.243.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":382,"threads_started":1}
W20260619 04:46:40.236483 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 12873: Not authorized: Client connection negotiation failed: client connection to 127.3.243.129:33653: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 04:46:40.236536 13335 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40364: Request ticket server kudu/127.3.243.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 04:46:40.236722 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 12873
I20260619 04:46:40.253352 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13024
I20260619 04:46:40.266966 13336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:40.254732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.3.243.130:38027)
0619 04:46:40.255477 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:40.255485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:40.255515 (+ 30us) server_negotiation.cc:407] Connection header received
0619 04:46:40.255630 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:40.255637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:40.255891 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:40.256108 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:40.257587 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.258681 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.259960 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.260675 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.260835 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:40.263498 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:40.263507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:40.263522 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:40.263564 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:40.266219 (+ 2655us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 04:46:40.266603 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35770: Request ticket server kudu/127.3.243.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":355,"threads_started":1}
W20260619 04:46:40.266968 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13024: Not authorized: Client connection negotiation failed: client connection to 127.3.243.130:38027: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 04:46:40.267074 13336 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35770: Request ticket server kudu/127.3.243.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 04:46:40.267107 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 13024
I20260619 04:46:40.283804 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13157
I20260619 04:46:40.296811 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:40.284918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.3.243.131:35319)
0619 04:46:40.285484 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:40.285491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:40.285518 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:46:40.285585 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:40.285591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:40.285829 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:40.286001 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:40.287807 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.288893 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.290126 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.290658 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.290752 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:40.293441 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:40.293450 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:40.293464 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:40.293549 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:40.296280 (+ 2731us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 04:46:40.296568 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39826: Request ticket server kudu/127.3.243.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":275,"threads_started":1}
W20260619 04:46:40.296931 13337 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39826: Request ticket server kudu/127.3.243.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 04:46:40.296952 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13157: Not authorized: Client connection negotiation failed: client connection to 127.3.243.131:35319: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.243.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 04:46:40.297117 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 13157
I20260619 04:46:40.313505 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 12804
I20260619 04:46:40.314795 12866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:40.434609 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 12804
2026-06-19T04:46:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21218 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T04:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:46:40Z Disabled control of system clock
I20260619 04:46:40.509805 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42347
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:42347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 04:46:40.834123 13342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:40.834409 13342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:40.834539 13342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:40.841753 13342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:40.841887 13342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:40.841996 13342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:40.842098 13342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:40.842182 13342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:40.851729 13342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:42347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:40.853078 13342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:40.854831 13342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:40.867197 13347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:40.867282 13348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:40.867395 13342 server_base.cc:1034] running on GCE node
W20260619 04:46:40.867576 13350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:40.868786 13342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:40.870131 13342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:40.871407 13342 hybrid_clock.cc:648] HybridClock initialized: now 1781844400871321 us; error 46 us; skew 500 ppm
I20260619 04:46:40.871790 13342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:46:40.874076 13342 webserver.cc:460] Webserver started at http://127.3.243.190:41381/ using document root <none> and password file <none>
I20260619 04:46:40.874778 13342 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:40.874910 13342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:40.875226 13342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:40.878122 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "a997c87cb2844f5989ec6f615aad6d32"
format_stamp: "Formatted at 2026-06-19 04:46:40 on dist-test-slave-2qp8"
server_key: "bfc37f3c54bb4c0b849eaa5b671b50cb"
server_key_iv: "1fe7eecda392bf3e3be3c406eee83e93"
server_key_version: "encryptionkey@0"
I20260619 04:46:40.878990 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "a997c87cb2844f5989ec6f615aad6d32"
format_stamp: "Formatted at 2026-06-19 04:46:40 on dist-test-slave-2qp8"
server_key: "bfc37f3c54bb4c0b849eaa5b671b50cb"
server_key_iv: "1fe7eecda392bf3e3be3c406eee83e93"
server_key_version: "encryptionkey@0"
I20260619 04:46:40.883939 13342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 04:46:40.887375 13356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:40.888746 13342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:46:40.888931 13342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "a997c87cb2844f5989ec6f615aad6d32"
format_stamp: "Formatted at 2026-06-19 04:46:40 on dist-test-slave-2qp8"
server_key: "bfc37f3c54bb4c0b849eaa5b671b50cb"
server_key_iv: "1fe7eecda392bf3e3be3c406eee83e93"
server_key_version: "encryptionkey@0"
I20260619 04:46:40.889130 13342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:40.907559 13342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:40.908398 13342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:40.908711 13342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:40.926893 13342 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:42347
I20260619 04:46:40.926911 13407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:42347 every 8 connection(s)
I20260619 04:46:40.928243 13342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:46:40.930847 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 13342
I20260619 04:46:40.931072 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:46:40.931864 4046 external_mini_cluster.cc:1428] Setting key 95e955167e916621aeb480714d317ae1
I20260619 04:46:40.932668 13408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:40.945345 13408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32: Bootstrap starting.
I20260619 04:46:40.945971 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:40.934032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.3.243.190:42347)
0619 04:46:40.935007 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:40.935018 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:40.935057 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:46:40.937339 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:40.937359 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:40.937803 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:40.938198 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:40.938802 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.940081 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.941479 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:40.942156 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:40.942406 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:40.943623 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:40.943631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:40.943644 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:40.944238 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:40.944316 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:40.944336 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:40.944462 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:40.944674 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 04:46:40.944971 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":346,"threads_started":1}
I20260619 04:46:40.948683 13408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:40.949703 13408 log.cc:824] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:40.952469 13408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32: No bootstrap required, opened a new log
I20260619 04:46:40.959760 13408 raft_consensus.cc:348] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a997c87cb2844f5989ec6f615aad6d32" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42347 } }
I20260619 04:46:40.960213 13408 raft_consensus.cc:374] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:40.960377 13408 raft_consensus.cc:729] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a997c87cb2844f5989ec6f615aad6d32, State: Initialized, Role: FOLLOWER
I20260619 04:46:40.961012 13408 consensus_queue.cc:260] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a997c87cb2844f5989ec6f615aad6d32" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42347 } }
I20260619 04:46:40.961287 13408 raft_consensus.cc:388] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:46:40.961480 13408 raft_consensus.cc:482] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:46:40.961681 13408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:40.964402 13408 raft_consensus.cc:504] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a997c87cb2844f5989ec6f615aad6d32" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42347 } }
I20260619 04:46:40.965078 13408 leader_election.cc:302] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a997c87cb2844f5989ec6f615aad6d32; no voters:
I20260619 04:46:40.965854 13408 leader_election.cc:288] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:46:40.965978 13413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:40.968611 13408 sys_catalog.cc:564] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:46:40.972175 13413 raft_consensus.cc:686] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [term 1 LEADER]: Becoming Leader. State: Replica: a997c87cb2844f5989ec6f615aad6d32, State: Running, Role: LEADER
I20260619 04:46:40.972779 13413 consensus_queue.cc:237] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a997c87cb2844f5989ec6f615aad6d32" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42347 } }
I20260619 04:46:40.976181 13415 sys_catalog.cc:455] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a997c87cb2844f5989ec6f615aad6d32. Latest consensus state: current_term: 1 leader_uuid: "a997c87cb2844f5989ec6f615aad6d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a997c87cb2844f5989ec6f615aad6d32" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42347 } } }
I20260619 04:46:40.976558 13415 sys_catalog.cc:458] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:40.976300 13414 sys_catalog.cc:455] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a997c87cb2844f5989ec6f615aad6d32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a997c87cb2844f5989ec6f615aad6d32" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42347 } } }
I20260619 04:46:40.976994 13414 sys_catalog.cc:458] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:40.979897 13423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:46:40.984952 13423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:46:40.993737 13423 catalog_manager.cc:1347] Generated new cluster ID: 90119a2ac8284d409d37a6f8f74d077a
I20260619 04:46:40.993856 13423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:46:41.008503 13423 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:46:41.009840 13423 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:46:41.025588 13423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a997c87cb2844f5989ec6f615aad6d32: Generated new TSK 0
I20260619 04:46:41.026535 13423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:46:41.034127 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42347
--builtin_ntp_servers=127.3.243.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 04:46:41.366184 13432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:41.366447 13432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:41.366616 13432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:41.373934 13432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:41.374079 13432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:41.374224 13432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:46:41.382982 13432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:41.384222 13432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:41.386015 13432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:41.398869 13438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:41.398897 13440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:41.398964 13437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:41.399603 13432 server_base.cc:1034] running on GCE node
I20260619 04:46:41.400240 13432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:41.401129 13432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:41.402391 13432 hybrid_clock.cc:648] HybridClock initialized: now 1781844401402341 us; error 52 us; skew 500 ppm
I20260619 04:46:41.402808 13432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:46:41.408422 13432 webserver.cc:460] Webserver started at http://127.3.243.129:35065/ using document root <none> and password file <none>
I20260619 04:46:41.409104 13432 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:41.409235 13432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:41.409569 13432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:41.412391 13432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "a7070a4199a54d93937e25a888f1eb9d"
format_stamp: "Formatted at 2026-06-19 04:46:41 on dist-test-slave-2qp8"
server_key: "f33db04309087d390306d8648bf616ca"
server_key_iv: "ee23380aa9e90a64e1da4f8bcd6f6533"
server_key_version: "encryptionkey@0"
I20260619 04:46:41.413247 13432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "a7070a4199a54d93937e25a888f1eb9d"
format_stamp: "Formatted at 2026-06-19 04:46:41 on dist-test-slave-2qp8"
server_key: "f33db04309087d390306d8648bf616ca"
server_key_iv: "ee23380aa9e90a64e1da4f8bcd6f6533"
server_key_version: "encryptionkey@0"
I20260619 04:46:41.418159 13432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:46:41.421386 13446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:41.422679 13432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:46:41.422848 13432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "a7070a4199a54d93937e25a888f1eb9d"
format_stamp: "Formatted at 2026-06-19 04:46:41 on dist-test-slave-2qp8"
server_key: "f33db04309087d390306d8648bf616ca"
server_key_iv: "ee23380aa9e90a64e1da4f8bcd6f6533"
server_key_version: "encryptionkey@0"
I20260619 04:46:41.423053 13432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:41.458832 13432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:41.459702 13432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:41.459992 13432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:41.460947 13432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:41.462615 13432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:41.462746 13432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:41.462867 13432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:41.462956 13432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:41.495847 13432 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:46443
I20260619 04:46:41.495942 13558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:46443 every 8 connection(s)
I20260619 04:46:41.497236 13432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:46:41.497850 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 13432
I20260619 04:46:41.498061 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:46:41.498960 4046 external_mini_cluster.cc:1428] Setting key d9179a6923225713292cf24ea1dc3ce0
I20260619 04:46:41.501858 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42347
--builtin_ntp_servers=127.3.243.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 04:46:41.510443 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:41.500215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:59577 (local address 127.3.243.190:42347)
0619 04:46:41.500667 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:41.500672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:41.502365 (+ 1693us) server_negotiation.cc:407] Connection header received
0619 04:46:41.504148 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:41.504154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:41.504244 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:41.504376 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:41.505414 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:41.506151 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:41.507651 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:41.508043 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:41.508209 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:41.509727 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:41.509732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:41.509735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:41.509751 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:41.509790 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:41.509803 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:41.509877 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:41.510113 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 04:46:41.510187 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":238,"threads_started":1}
I20260619 04:46:41.511915 13562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:41.500940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42347 (local address 127.3.243.129:59577)
0619 04:46:41.502204 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:41.502255 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:41.503850 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:41.504603 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:41.504612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:41.504699 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:41.505249 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:41.505265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:41.506296 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:41.506302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:41.507509 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:41.507520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:41.508309 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:41.508317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:41.508642 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:41.509380 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:46:41.509386 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:41.509436 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:46:41.509536 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:41.509901 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:41.509913 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:41.509924 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:46:41.510098 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 04:46:41.510502 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":352,"threads_started":1}
I20260619 04:46:41.513489 13559 heartbeater.cc:347] Connected to a master server at 127.3.243.190:42347
I20260619 04:46:41.513846 13559 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:41.514681 13559 heartbeater.cc:511] Master 127.3.243.190:42347 requested a full tablet report, sending...
I20260619 04:46:41.516685 13373 ts_manager.cc:194] Registered new tserver with Master: a7070a4199a54d93937e25a888f1eb9d (127.3.243.129:46443)
I20260619 04:46:41.518246 13373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.129:59577
W20260619 04:46:41.819469 13563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:41.819766 13563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:41.819923 13563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:41.827212 13563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:41.827342 13563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:41.827556 13563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:46:41.836412 13563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:41.837661 13563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:41.839382 13563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:41.852205 13568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:41.852185 13569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:41.852339 13571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:41.852950 13563 server_base.cc:1034] running on GCE node
I20260619 04:46:41.853514 13563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:41.854403 13563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:41.855626 13563 hybrid_clock.cc:648] HybridClock initialized: now 1781844401855585 us; error 55 us; skew 500 ppm
I20260619 04:46:41.855947 13563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:46:41.858202 13563 webserver.cc:460] Webserver started at http://127.3.243.130:37483/ using document root <none> and password file <none>
I20260619 04:46:41.858880 13563 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:41.859048 13563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:41.859355 13563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:41.862107 13563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "8d685c532e434523ad6abc30cd072c22"
format_stamp: "Formatted at 2026-06-19 04:46:41 on dist-test-slave-2qp8"
server_key: "48192ad502a28168900a905067171c9c"
server_key_iv: "69583f82f660a64e2ac17b4609cb1e53"
server_key_version: "encryptionkey@0"
I20260619 04:46:41.863023 13563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "8d685c532e434523ad6abc30cd072c22"
format_stamp: "Formatted at 2026-06-19 04:46:41 on dist-test-slave-2qp8"
server_key: "48192ad502a28168900a905067171c9c"
server_key_iv: "69583f82f660a64e2ac17b4609cb1e53"
server_key_version: "encryptionkey@0"
I20260619 04:46:41.867924 13563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 04:46:41.871336 13577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:41.872563 13563 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:46:41.872723 13563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "8d685c532e434523ad6abc30cd072c22"
format_stamp: "Formatted at 2026-06-19 04:46:41 on dist-test-slave-2qp8"
server_key: "48192ad502a28168900a905067171c9c"
server_key_iv: "69583f82f660a64e2ac17b4609cb1e53"
server_key_version: "encryptionkey@0"
I20260619 04:46:41.872932 13563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:41.903052 13563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:41.903918 13563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:41.904350 13563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:41.905323 13563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:41.906720 13563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:41.906840 13563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:41.906981 13563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:41.907068 13563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:41.940282 13563 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:44993
I20260619 04:46:41.940325 13689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:44993 every 8 connection(s)
I20260619 04:46:41.941538 13563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:46:41.946180 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 13563
I20260619 04:46:41.946334 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:46:41.946795 4046 external_mini_cluster.cc:1428] Setting key 623300ff2888ab42ba20ba7a4d3d36b6
I20260619 04:46:41.949505 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42347
--builtin_ntp_servers=127.3.243.148:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 04:46:41.954324 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:41.944244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:35159 (local address 127.3.243.190:42347)
0619 04:46:41.944436 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:41.944441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:41.945876 (+ 1435us) server_negotiation.cc:407] Connection header received
0619 04:46:41.947968 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:41.947973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:41.948061 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:41.948203 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:41.949340 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:41.950143 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:41.951590 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:41.951990 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:41.952158 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:41.953551 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:41.953555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:41.953558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:41.953574 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:41.953604 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:41.953615 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:41.953709 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:41.953939 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 04:46:41.954027 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:46:41.955263 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:41.944869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42347 (local address 127.3.243.130:35159)
0619 04:46:41.945719 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:41.945758 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:41.947704 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:41.948451 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:41.948462 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:41.948577 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:41.949186 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:41.949203 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:41.950278 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:41.950283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:41.951451 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:41.951462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:41.952231 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:41.952238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:41.952521 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:41.953236 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:46:41.953239 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:41.953277 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:46:41.953346 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:41.953710 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:41.953722 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:41.953732 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 04:46:41.953904 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 04:46:41.954271 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":252,"threads_started":1}
I20260619 04:46:41.956710 13690 heartbeater.cc:347] Connected to a master server at 127.3.243.190:42347
I20260619 04:46:41.956998 13690 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:41.957718 13690 heartbeater.cc:511] Master 127.3.243.190:42347 requested a full tablet report, sending...
I20260619 04:46:41.959120 13373 ts_manager.cc:194] Registered new tserver with Master: 8d685c532e434523ad6abc30cd072c22 (127.3.243.130:44993)
I20260619 04:46:41.960355 13373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.130:35159
W20260619 04:46:42.273644 13693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:42.273983 13693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:42.274080 13693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:42.281774 13693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:42.281869 13693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:42.281981 13693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:46:42.290673 13693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:42.291899 13693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:42.293756 13693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:42.306010 13698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:42.306012 13701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:42.306044 13699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:42.306574 13693 server_base.cc:1034] running on GCE node
I20260619 04:46:42.307199 13693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:42.308086 13693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:42.309363 13693 hybrid_clock.cc:648] HybridClock initialized: now 1781844402309306 us; error 58 us; skew 500 ppm
I20260619 04:46:42.309651 13693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:46:42.311717 13693 webserver.cc:460] Webserver started at http://127.3.243.131:42117/ using document root <none> and password file <none>
I20260619 04:46:42.312443 13693 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:42.312570 13693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:42.312879 13693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:42.315640 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "9673cb1c15cd45168dfa6e68f60ee9e3"
format_stamp: "Formatted at 2026-06-19 04:46:42 on dist-test-slave-2qp8"
server_key: "1fb5d425cd2a3cf98aaff4d3060f6bee"
server_key_iv: "cddec6e82df8fccc58775425c78efa9b"
server_key_version: "encryptionkey@0"
I20260619 04:46:42.316447 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "9673cb1c15cd45168dfa6e68f60ee9e3"
format_stamp: "Formatted at 2026-06-19 04:46:42 on dist-test-slave-2qp8"
server_key: "1fb5d425cd2a3cf98aaff4d3060f6bee"
server_key_iv: "cddec6e82df8fccc58775425c78efa9b"
server_key_version: "encryptionkey@0"
I20260619 04:46:42.321192 13693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 04:46:42.324532 13707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:42.325757 13693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:46:42.325933 13693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "9673cb1c15cd45168dfa6e68f60ee9e3"
format_stamp: "Formatted at 2026-06-19 04:46:42 on dist-test-slave-2qp8"
server_key: "1fb5d425cd2a3cf98aaff4d3060f6bee"
server_key_iv: "cddec6e82df8fccc58775425c78efa9b"
server_key_version: "encryptionkey@0"
I20260619 04:46:42.326140 13693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:42.343822 13693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:42.344744 13693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:42.345053 13693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:42.345978 13693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:42.347330 13693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:42.347460 13693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:42.347599 13693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:42.347687 13693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:42.379643 13693 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:35125
I20260619 04:46:42.379686 13819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:35125 every 8 connection(s)
I20260619 04:46:42.381004 13693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:46:42.382618 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 13693
I20260619 04:46:42.382781 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:46:42.383219 4046 external_mini_cluster.cc:1428] Setting key 359ffe0fe70016d3a085def92c2541c4
I20260619 04:46:42.393168 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:42.383539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:46327 (local address 127.3.243.190:42347)
0619 04:46:42.383746 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:42.383752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:42.385697 (+ 1945us) server_negotiation.cc:407] Connection header received
0619 04:46:42.387417 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:42.387421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:42.387485 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:42.387561 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:42.388606 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.389316 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.390712 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.391107 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.391193 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:42.392467 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:42.392471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:42.392474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:42.392490 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:42.392519 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:42.392530 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:42.392592 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:42.392828 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 04:46:42.392900 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:46:42.394132 13822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:42.384325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42347 (local address 127.3.243.131:46327)
0619 04:46:42.385539 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:42.385584 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:42.387162 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:42.387730 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:42.387738 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:42.387821 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:42.388463 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:42.388480 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.389449 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.389452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:42.390599 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:42.390613 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.391276 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.391280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:42.391465 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:42.392078 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:46:42.392080 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:42.392162 (+ 82us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:46:42.392286 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:42.392606 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:42.392612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:42.392617 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:46:42.392797 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 04:46:42.393084 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":469,"threads_started":1}
I20260619 04:46:42.395437 13820 heartbeater.cc:347] Connected to a master server at 127.3.243.190:42347
I20260619 04:46:42.395716 13820 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:42.396489 13820 heartbeater.cc:511] Master 127.3.243.190:42347 requested a full tablet report, sending...
I20260619 04:46:42.397843 13373 ts_manager.cc:194] Registered new tserver with Master: 9673cb1c15cd45168dfa6e68f60ee9e3 (127.3.243.131:35125)
I20260619 04:46:42.398568 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:46:42.398669 13373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.131:46327
W20260619 04:46:42.404512 13561 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:42.402947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.3.243.190:42347)
0619 04:46:42.403105 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:42.403109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:42.403592 (+ 483us) server_negotiation.cc:407] Connection header received
0619 04:46:42.403716 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:42.403719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:42.403786 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:42.403863 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:42.404255 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41432: BlockingRecv error: recv got EOF from 127.0.0.1:41432 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:46:42.405359 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13432
I20260619 04:46:42.413009 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:42.406277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.3.243.129:46443)
0619 04:46:42.406780 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:42.406787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:42.406808 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:46:42.406860 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:42.406865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:42.407038 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:42.407192 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:42.407575 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.408586 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.409886 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.410442 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.410580 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:42.411657 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:42.411664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:42.411675 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:42.412257 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:42.412298 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:42.412316 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:42.412438 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:42.412581 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 04:46:42.412704 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260619 04:46:42.414143 13554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:42.495730 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 13432
I20260619 04:46:42.512770 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13563
I20260619 04:46:42.520352 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:42.513696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56024 (local address 127.3.243.130:44993)
0619 04:46:42.514171 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:42.514178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:42.514204 (+ 26us) server_negotiation.cc:407] Connection header received
0619 04:46:42.514261 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:42.514267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:42.514491 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:42.514611 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:42.514965 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.516087 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.517338 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.517869 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.517965 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:42.518878 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:42.518886 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:42.518900 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:42.519484 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:42.519532 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:42.519552 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:42.519652 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:42.519759 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 04:46:42.519891 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":225,"threads_started":1}
I20260619 04:46:42.521588 13685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:42.603754 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 13563
I20260619 04:46:42.620241 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13693
I20260619 04:46:42.627985 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:42.621208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.3.243.131:35125)
0619 04:46:42.621869 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:42.621882 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:42.621928 (+ 46us) server_negotiation.cc:407] Connection header received
0619 04:46:42.621989 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:42.621996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:42.622231 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:42.622381 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:42.622723 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.623723 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.624967 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:42.625507 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:42.625602 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:42.626568 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:42.626576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:42.626590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:42.627177 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:42.627222 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:42.627241 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:42.627326 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:42.627451 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 04:46:42.627613 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":562,"threads_started":1}
I20260619 04:46:42.629232 13815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:42.710800 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 13693
I20260619 04:46:42.726572 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13342
I20260619 04:46:42.727769 13403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:42.831960 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 13342
2026-06-19T04:46:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2392 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T04:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:46:42Z Disabled control of system clock
I20260619 04:46:42.901620 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45739
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:45739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 04:46:43.227645 13839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:43.227926 13839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:43.228006 13839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:43.235164 13839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:43.235280 13839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:43.235320 13839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:43.235369 13839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:43.235441 13839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:43.244490 13839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:45739
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:43.245738 13839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:43.247433 13839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:43.259083 13844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:43.259159 13845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:43.259137 13847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:43.259919 13839 server_base.cc:1034] running on GCE node
I20260619 04:46:43.260792 13839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:43.262233 13839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:43.263485 13839 hybrid_clock.cc:648] HybridClock initialized: now 1781844403263444 us; error 65 us; skew 500 ppm
I20260619 04:46:43.263833 13839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:46:43.266100 13839 webserver.cc:460] Webserver started at http://127.3.243.190:34807/ using document root <none> and password file <none>
I20260619 04:46:43.266795 13839 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:43.266927 13839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:43.267236 13839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:43.270015 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "486668f8c2944ec386b3918411ecdf51"
format_stamp: "Formatted at 2026-06-19 04:46:43 on dist-test-slave-2qp8"
server_key: "c3ae4cc5b90fb174a0025f80cdd68dad"
server_key_iv: "32af030ecb76f119f30590633825c42c"
server_key_version: "encryptionkey@0"
I20260619 04:46:43.270761 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "486668f8c2944ec386b3918411ecdf51"
format_stamp: "Formatted at 2026-06-19 04:46:43 on dist-test-slave-2qp8"
server_key: "c3ae4cc5b90fb174a0025f80cdd68dad"
server_key_iv: "32af030ecb76f119f30590633825c42c"
server_key_version: "encryptionkey@0"
I20260619 04:46:43.275575 13839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 04:46:43.278831 13853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:43.279973 13839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 04:46:43.280184 13839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "486668f8c2944ec386b3918411ecdf51"
format_stamp: "Formatted at 2026-06-19 04:46:43 on dist-test-slave-2qp8"
server_key: "c3ae4cc5b90fb174a0025f80cdd68dad"
server_key_iv: "32af030ecb76f119f30590633825c42c"
server_key_version: "encryptionkey@0"
I20260619 04:46:43.280459 13839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:43.285952 13839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:43.286676 13839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:43.286979 13839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:43.304947 13839 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:45739
I20260619 04:46:43.304991 13904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:45739 every 8 connection(s)
I20260619 04:46:43.306314 13839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:46:43.309948 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:43.311220 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 13839
I20260619 04:46:43.311465 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:46:43.312309 4046 external_mini_cluster.cc:1428] Setting key e98466ef93259b5e8a2875aae7fca787
I20260619 04:46:43.320643 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:43.314376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.3.243.190:45739)
0619 04:46:43.315393 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:43.315403 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:43.315442 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:46:43.317078 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:43.317109 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:43.317533 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:43.317858 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:43.318915 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:43.318920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:43.318932 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:43.318946 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:43.319002 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:43.319018 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:43.319174 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:43.319304 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 04:46:43.319611 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":391,"threads_started":1}
I20260619 04:46:43.323859 13905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51: Bootstrap starting.
I20260619 04:46:43.326737 13905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:43.327700 13905 log.cc:824] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:43.330440 13905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51: No bootstrap required, opened a new log
I20260619 04:46:43.337826 13905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486668f8c2944ec386b3918411ecdf51" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45739 } }
I20260619 04:46:43.338167 13905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:43.338259 13905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486668f8c2944ec386b3918411ecdf51, State: Initialized, Role: FOLLOWER
I20260619 04:46:43.338792 13905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486668f8c2944ec386b3918411ecdf51" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45739 } }
I20260619 04:46:43.339025 13905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:46:43.339202 13905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:46:43.339382 13905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:43.341676 13905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486668f8c2944ec386b3918411ecdf51" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45739 } }
I20260619 04:46:43.342199 13905 leader_election.cc:302] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486668f8c2944ec386b3918411ecdf51; no voters:
I20260619 04:46:43.342800 13905 leader_election.cc:288] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:46:43.342962 13910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:43.343695 13910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [term 1 LEADER]: Becoming Leader. State: Replica: 486668f8c2944ec386b3918411ecdf51, State: Running, Role: LEADER
I20260619 04:46:43.344303 13910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486668f8c2944ec386b3918411ecdf51" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45739 } }
I20260619 04:46:43.344970 13905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:46:43.346800 13911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486668f8c2944ec386b3918411ecdf51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486668f8c2944ec386b3918411ecdf51" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45739 } } }
I20260619 04:46:43.347499 13911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:43.347455 13912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486668f8c2944ec386b3918411ecdf51. Latest consensus state: current_term: 1 leader_uuid: "486668f8c2944ec386b3918411ecdf51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486668f8c2944ec386b3918411ecdf51" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45739 } } }
I20260619 04:46:43.352422 13912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:43.356758 13917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:46:43.361660 13917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:46:43.371601 13917 catalog_manager.cc:1347] Generated new cluster ID: 50366b046ea2496088a28838f6e49287
I20260619 04:46:43.371730 13917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:46:43.396610 13917 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:46:43.397058 13917 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:46:43.404093 13917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486668f8c2944ec386b3918411ecdf51: Generated new TSK 0
I20260619 04:46:43.404947 13917 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:46:43.415999 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45739
--builtin_ntp_servers=127.3.243.148:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 04:46:43.743259 13929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:43.743534 13929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:43.743621 13929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:43.751052 13929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:43.751148 13929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:43.751256 13929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:46:43.760875 13929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:43.762252 13929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:43.763940 13929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:43.777218 13934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:43.777273 13935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:43.777297 13937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:43.778090 13929 server_base.cc:1034] running on GCE node
I20260619 04:46:43.778712 13929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:43.779567 13929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:43.780753 13929 hybrid_clock.cc:648] HybridClock initialized: now 1781844403780684 us; error 61 us; skew 500 ppm
I20260619 04:46:43.781075 13929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:46:43.783245 13929 webserver.cc:460] Webserver started at http://127.3.243.129:45453/ using document root <none> and password file <none>
I20260619 04:46:43.784040 13929 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:43.784183 13929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:43.784539 13929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:43.788750 13929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "70de2038e81c4edda73e8716888769da"
format_stamp: "Formatted at 2026-06-19 04:46:43 on dist-test-slave-2qp8"
server_key: "b79ade3f7998cb584ef1d6d9325771b8"
server_key_iv: "53cc5cdd4cc58964fb2be8e0889f1f95"
server_key_version: "encryptionkey@0"
I20260619 04:46:43.789534 13929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "70de2038e81c4edda73e8716888769da"
format_stamp: "Formatted at 2026-06-19 04:46:43 on dist-test-slave-2qp8"
server_key: "b79ade3f7998cb584ef1d6d9325771b8"
server_key_iv: "53cc5cdd4cc58964fb2be8e0889f1f95"
server_key_version: "encryptionkey@0"
I20260619 04:46:43.794471 13929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:46:43.797914 13943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:43.799140 13929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:46:43.799309 13929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "70de2038e81c4edda73e8716888769da"
format_stamp: "Formatted at 2026-06-19 04:46:43 on dist-test-slave-2qp8"
server_key: "b79ade3f7998cb584ef1d6d9325771b8"
server_key_iv: "53cc5cdd4cc58964fb2be8e0889f1f95"
server_key_version: "encryptionkey@0"
I20260619 04:46:43.799530 13929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:43.804993 13929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:43.805753 13929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:43.806020 13929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:43.806897 13929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:43.808244 13929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:43.808363 13929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:43.808506 13929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:43.808593 13929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:43.841091 13929 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:42421
I20260619 04:46:43.841140 14055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:42421 every 8 connection(s)
I20260619 04:46:43.842423 13929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:46:43.848649 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 13929
I20260619 04:46:43.848871 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:46:43.849668 4046 external_mini_cluster.cc:1428] Setting key 9db0f41553b2e17264dbfcf3187d5b92
I20260619 04:46:43.850941 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:43.845001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:46989 (local address 127.3.243.190:45739)
0619 04:46:43.845517 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:43.845523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:43.846640 (+ 1117us) server_negotiation.cc:407] Connection header received
0619 04:46:43.848515 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:43.848519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:43.848569 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:43.848657 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:43.850164 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:43.850167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:43.850170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:43.850186 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:43.850249 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:43.850267 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:43.850388 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:43.850582 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 04:46:43.850659 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":309,"threads_started":1}
I20260619 04:46:43.852017 14059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:43.845503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45739 (local address 127.3.243.129:46989)
0619 04:46:43.846478 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:43.846524 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:43.848102 (+ 1578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:43.848962 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:43.848972 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:43.849099 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:43.849810 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:46:43.849815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:43.849867 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:46:43.849974 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:43.850372 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:43.850381 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:43.850386 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:46:43.850656 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 04:46:43.850915 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":308,"threads_started":1}
I20260619 04:46:43.852862 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45739
--builtin_ntp_servers=127.3.243.148:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 04:46:43.853632 14056 heartbeater.cc:347] Connected to a master server at 127.3.243.190:45739
I20260619 04:46:43.853946 14056 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:43.854599 14056 heartbeater.cc:511] Master 127.3.243.190:45739 requested a full tablet report, sending...
I20260619 04:46:43.856884 13870 ts_manager.cc:194] Registered new tserver with Master: 70de2038e81c4edda73e8716888769da (127.3.243.129:42421)
W20260619 04:46:44.182673 14060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:44.183159 14060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:44.183338 14060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:44.192999 14060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:44.193120 14060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:44.193392 14060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:46:44.203001 14060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:44.204523 14060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:44.206475 14060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:44.219798 14065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:44.219748 14066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:44.219776 14068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:44.220661 14060 server_base.cc:1034] running on GCE node
I20260619 04:46:44.221213 14060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:44.222144 14060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:44.223335 14060 hybrid_clock.cc:648] HybridClock initialized: now 1781844404223272 us; error 62 us; skew 500 ppm
I20260619 04:46:44.223614 14060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:46:44.226053 14060 webserver.cc:460] Webserver started at http://127.3.243.130:39309/ using document root <none> and password file <none>
I20260619 04:46:44.226701 14060 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:44.226780 14060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:44.227041 14060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:44.229805 14060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "78012fbba9e9462b9a4a2dad127c2207"
format_stamp: "Formatted at 2026-06-19 04:46:44 on dist-test-slave-2qp8"
server_key: "f42c7036380e1c09fe1c8c9b02103360"
server_key_iv: "27e04455b35cc8bfd83e2502de2b4e26"
server_key_version: "encryptionkey@0"
I20260619 04:46:44.230526 14060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "78012fbba9e9462b9a4a2dad127c2207"
format_stamp: "Formatted at 2026-06-19 04:46:44 on dist-test-slave-2qp8"
server_key: "f42c7036380e1c09fe1c8c9b02103360"
server_key_iv: "27e04455b35cc8bfd83e2502de2b4e26"
server_key_version: "encryptionkey@0"
I20260619 04:46:44.235307 14060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 04:46:44.238567 14074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:44.239825 14060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:46:44.239997 14060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "78012fbba9e9462b9a4a2dad127c2207"
format_stamp: "Formatted at 2026-06-19 04:46:44 on dist-test-slave-2qp8"
server_key: "f42c7036380e1c09fe1c8c9b02103360"
server_key_iv: "27e04455b35cc8bfd83e2502de2b4e26"
server_key_version: "encryptionkey@0"
I20260619 04:46:44.240264 14060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:44.245646 14060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:44.246474 14060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:44.246734 14060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:44.247611 14060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:44.249047 14060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:44.249166 14060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:44.249294 14060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:44.249380 14060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:44.281287 14060 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:46223
I20260619 04:46:44.281332 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:46223 every 8 connection(s)
I20260619 04:46:44.282599 14060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:46:44.287026 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 14060
I20260619 04:46:44.287218 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:46:44.287771 4046 external_mini_cluster.cc:1428] Setting key de065a1c12243623d436a6b1283a194a
I20260619 04:46:44.290860 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45739
--builtin_ntp_servers=127.3.243.148:36061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 04:46:44.292070 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:44.285310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:55101 (local address 127.3.243.190:45739)
0619 04:46:44.285501 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:44.285506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:44.287360 (+ 1854us) server_negotiation.cc:407] Connection header received
0619 04:46:44.289540 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:44.289548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:44.289642 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:44.289741 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:44.291226 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:44.291235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:44.291242 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:44.291265 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:44.291306 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:44.291323 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:44.291448 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:44.291624 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 04:46:44.291691 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:46:44.292863 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:44.286044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45739 (local address 127.3.243.130:55101)
0619 04:46:44.287173 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:44.287224 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:44.289207 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:44.290021 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:44.290029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:44.290120 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:44.290885 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:46:44.290887 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:44.290925 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:46:44.291036 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:44.291427 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:44.291434 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:44.291440 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:46:44.291648 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 04:46:44.291843 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":312,"threads_started":1}
I20260619 04:46:44.294221 14187 heartbeater.cc:347] Connected to a master server at 127.3.243.190:45739
I20260619 04:46:44.294523 14187 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:44.295073 14187 heartbeater.cc:511] Master 127.3.243.190:45739 requested a full tablet report, sending...
I20260619 04:46:44.296430 13870 ts_manager.cc:194] Registered new tserver with Master: 78012fbba9e9462b9a4a2dad127c2207 (127.3.243.130:46223)
W20260619 04:46:44.614634 14190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:44.614892 14190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:44.615068 14190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:44.622520 14190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:44.622611 14190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:44.622782 14190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:46:44.631793 14190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:36061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45739
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:44.633085 14190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:44.634820 14190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:44.647957 14196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:44.647982 14198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:44.648043 14195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:44.649009 14190 server_base.cc:1034] running on GCE node
I20260619 04:46:44.649700 14190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:44.650707 14190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:44.651938 14190 hybrid_clock.cc:648] HybridClock initialized: now 1781844404651895 us; error 58 us; skew 500 ppm
I20260619 04:46:44.652370 14190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:46:44.654614 14190 webserver.cc:460] Webserver started at http://127.3.243.131:40599/ using document root <none> and password file <none>
I20260619 04:46:44.655402 14190 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:44.655534 14190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:44.655870 14190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:44.658730 14190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "7339cdf0ea0c4934bb9a9deea907e46c"
format_stamp: "Formatted at 2026-06-19 04:46:44 on dist-test-slave-2qp8"
server_key: "f0d0a61826c1d77951364871e659e582"
server_key_iv: "4c74b91247eba856c33d661eb442d883"
server_key_version: "encryptionkey@0"
I20260619 04:46:44.659523 14190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "7339cdf0ea0c4934bb9a9deea907e46c"
format_stamp: "Formatted at 2026-06-19 04:46:44 on dist-test-slave-2qp8"
server_key: "f0d0a61826c1d77951364871e659e582"
server_key_iv: "4c74b91247eba856c33d661eb442d883"
server_key_version: "encryptionkey@0"
I20260619 04:46:44.664503 14190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260619 04:46:44.667845 14204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:44.669167 14190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:46:44.669353 14190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "7339cdf0ea0c4934bb9a9deea907e46c"
format_stamp: "Formatted at 2026-06-19 04:46:44 on dist-test-slave-2qp8"
server_key: "f0d0a61826c1d77951364871e659e582"
server_key_iv: "4c74b91247eba856c33d661eb442d883"
server_key_version: "encryptionkey@0"
I20260619 04:46:44.669569 14190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:44.675386 14190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:44.676258 14190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:44.676522 14190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:44.677367 14190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:44.678627 14190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:44.678745 14190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:44.678875 14190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:44.678959 14190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:44.711187 14190 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:39537
I20260619 04:46:44.711241 14316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:39537 every 8 connection(s)
I20260619 04:46:44.712543 14190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:46:44.713635 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 14190
I20260619 04:46:44.713824 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:46:44.714385 4046 external_mini_cluster.cc:1428] Setting key dafa8c320cebfd537b1c625bcc73cfa8
I20260619 04:46:44.721597 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:44.715442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:59905 (local address 127.3.243.190:45739)
0619 04:46:44.715626 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:44.715633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:44.717508 (+ 1875us) server_negotiation.cc:407] Connection header received
0619 04:46:44.719247 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:44.719253 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:44.719318 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:44.719439 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:44.720850 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:44.720856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:44.720861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:44.720881 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:44.720928 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:44.720940 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:44.721076 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:44.721257 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 04:46:44.721327 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:46:44.722572 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:44.716141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45739 (local address 127.3.243.131:59905)
0619 04:46:44.717344 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:44.717391 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:44.718994 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:44.719679 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:44.719687 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:44.719781 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:44.720555 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:46:44.720558 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:44.720596 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:46:44.720681 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:44.721037 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:44.721047 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:44.721057 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 04:46:44.721341 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 04:46:44.721621 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":384,"threads_started":1}
I20260619 04:46:44.723933 14317 heartbeater.cc:347] Connected to a master server at 127.3.243.190:45739
I20260619 04:46:44.724265 14317 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:44.724820 14317 heartbeater.cc:511] Master 127.3.243.190:45739 requested a full tablet report, sending...
I20260619 04:46:44.726135 13870 ts_manager.cc:194] Registered new tserver with Master: 7339cdf0ea0c4934bb9a9deea907e46c (127.3.243.131:39537)
I20260619 04:46:44.729897 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 04:46:44.735661 14058 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:46:44.734212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.3.243.190:45739)
0619 04:46:44.734357 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:44.734361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:44.734863 (+ 502us) server_negotiation.cc:407] Connection header received
0619 04:46:44.735026 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:44.735029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:44.735076 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:44.735157 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:44.735417 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60584: BlockingRecv error: recv got EOF from 127.0.0.1:60584 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:46:44.736562 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13929
I20260619 04:46:44.739667 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:44.737428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.3.243.129:42421)
0619 04:46:44.737917 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:44.737924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:44.737949 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:46:44.738007 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:44.738012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:44.738205 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:44.738322 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:44.739132 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:44.739139 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:44.739150 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:44.739161 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:44.739198 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:44.739211 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:44.739315 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:44.739383 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 04:46:44.739462 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":237,"threads_started":1}
I20260619 04:46:44.740906 14051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:44.821246 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 13929
I20260619 04:46:44.837859 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 14060
I20260619 04:46:44.841665 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:44.838895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.3.243.130:46223)
0619 04:46:44.839430 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:44.839444 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:44.839469 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:46:44.839528 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:44.839534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:44.839752 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:44.839874 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:44.840860 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:44.840868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:44.840880 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:44.840902 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:44.840953 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:44.840969 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:44.841078 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:44.841236 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 04:46:44.841369 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":220,"threads_started":1}
I20260619 04:46:44.842918 14182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:44.922972 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 14060
I20260619 04:46:44.938529 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 14190
I20260619 04:46:44.942592 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:44.939488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.3.243.131:39537)
0619 04:46:44.940100 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:44.940149 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:44.940192 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:46:44.940261 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:44.940267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:44.940493 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:44.940662 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:44.941836 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:44.941842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:44.941854 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:46:44.941870 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:44.941909 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:46:44.941922 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:44.942104 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:44.942179 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 04:46:44.942284 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":305,"threads_started":1}
I20260619 04:46:44.943907 14312 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:45.024837 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 14190
I20260619 04:46:45.040750 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 13839
I20260619 04:46:45.042114 13900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:45.144474 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 13839
2026-06-19T04:46:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2313 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:46:45 dist-test-slave-2qp8 krb5kdc[14335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:46:45 dist-test-slave-2qp8 krb5kdc[14335](info): set up 2 sockets
Jun 19 04:46:45 dist-test-slave-2qp8 krb5kdc[14335](info): commencing operation
krb5kdc: starting...
W20260619 04:46:46.241254 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:46:46 dist-test-slave-2qp8 krb5kdc[14335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:46:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:46:46.471500 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44349
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:46.789496 14348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:46.789774 14348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:46.789861 14348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:46.796947 14348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:46:46.797046 14348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:46.797093 14348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:46.797150 14348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:46:46.797192 14348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:46:46.807793 14348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:46.809083 14348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:46.810271 14348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:46.821869 14353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:46.822013 14354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:46.822181 14356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:46.823032 14348 server_base.cc:1034] running on GCE node
I20260619 04:46:46.823750 14348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:46.825145 14348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:46.826367 14348 hybrid_clock.cc:648] HybridClock initialized: now 1781844406826319 us; error 47 us; skew 500 ppm
I20260619 04:46:46.826702 14348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:46 dist-test-slave-2qp8 krb5kdc[14335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:46.834268 14348 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:46:46.835760 14348 webserver.cc:460] Webserver started at http://127.3.243.190:36611/ using document root <none> and password file <none>
I20260619 04:46:46.836468 14348 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:46.836568 14348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:46.836812 14348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:46.839617 14348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "a878cdea1c924a35a02b84e72f429493"
format_stamp: "Formatted at 2026-06-19 04:46:46 on dist-test-slave-2qp8"
server_key: "c0e16b74296da7e54fe61da90c7303a4"
server_key_iv: "f3fd6d5356aca1b0425c5e01f57d5f20"
server_key_version: "encryptionkey@0"
I20260619 04:46:46.840413 14348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "a878cdea1c924a35a02b84e72f429493"
format_stamp: "Formatted at 2026-06-19 04:46:46 on dist-test-slave-2qp8"
server_key: "c0e16b74296da7e54fe61da90c7303a4"
server_key_iv: "f3fd6d5356aca1b0425c5e01f57d5f20"
server_key_version: "encryptionkey@0"
I20260619 04:46:46.845233 14348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:46:46.848453 14363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:46.849684 14348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:46:46.849856 14348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "a878cdea1c924a35a02b84e72f429493"
format_stamp: "Formatted at 2026-06-19 04:46:46 on dist-test-slave-2qp8"
server_key: "c0e16b74296da7e54fe61da90c7303a4"
server_key_iv: "f3fd6d5356aca1b0425c5e01f57d5f20"
server_key_version: "encryptionkey@0"
I20260619 04:46:46.850049 14348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:46.882287 14348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:46.886394 14348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:46.886626 14348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:46.904599 14348 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44349
I20260619 04:46:46.904623 14414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44349 every 8 connection(s)
I20260619 04:46:46.905949 14348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:46:46.909579 14415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:46.911751 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 14348
I20260619 04:46:46.912001 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:46:46.912793 4046 external_mini_cluster.cc:1428] Setting key eacb415e03478dcf65cc37832659298e
I20260619 04:46:46.922257 14415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493: Bootstrap starting.
I20260619 04:46:46.926093 14415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:46.927358 14415 log.cc:824] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493: Log is configured to *not* fsync() on all Append() calls
Jun 19 04:46:46 dist-test-slave-2qp8 krb5kdc[14335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:46:46.930248 14415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493: No bootstrap required, opened a new log
I20260619 04:46:46.936244 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:46.914879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.3.243.190:44349)
0619 04:46:46.915784 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:46.915796 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:46.915837 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:46:46.917536 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:46.917558 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:46.917944 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:46.918260 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:46.920017 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:46.921348 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:46.923138 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:46.923825 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:46.923967 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:46.928999 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:46.929007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:46.929022 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:46.929055 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:46.932085 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:46.932830 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:46.932836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:46.932842 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:46.933073 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:46.933473 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:46.933478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:46.933483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:46.934001 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:46.934359 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:46.934723 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 04:46:46.935159 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":476,"threads_started":1}
I20260619 04:46:46.938144 14415 raft_consensus.cc:348] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a878cdea1c924a35a02b84e72f429493" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44349 } }
I20260619 04:46:46.938515 14415 raft_consensus.cc:374] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:46.938632 14415 raft_consensus.cc:729] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a878cdea1c924a35a02b84e72f429493, State: Initialized, Role: FOLLOWER
I20260619 04:46:46.939248 14415 consensus_queue.cc:260] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a878cdea1c924a35a02b84e72f429493" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44349 } }
I20260619 04:46:46.939496 14415 raft_consensus.cc:388] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:46:46.939635 14415 raft_consensus.cc:482] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:46:46.939810 14415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:46.942307 14415 raft_consensus.cc:504] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a878cdea1c924a35a02b84e72f429493" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44349 } }
I20260619 04:46:46.942785 14415 leader_election.cc:302] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a878cdea1c924a35a02b84e72f429493; no voters:
I20260619 04:46:46.943352 14415 leader_election.cc:288] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:46:46.943470 14420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:46.944198 14420 raft_consensus.cc:686] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [term 1 LEADER]: Becoming Leader. State: Replica: a878cdea1c924a35a02b84e72f429493, State: Running, Role: LEADER
I20260619 04:46:46.944722 14420 consensus_queue.cc:237] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a878cdea1c924a35a02b84e72f429493" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44349 } }
I20260619 04:46:46.945438 14415 sys_catalog.cc:564] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:46:46.947666 14421 sys_catalog.cc:455] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a878cdea1c924a35a02b84e72f429493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a878cdea1c924a35a02b84e72f429493" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44349 } } }
I20260619 04:46:46.947695 14422 sys_catalog.cc:455] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a878cdea1c924a35a02b84e72f429493. Latest consensus state: current_term: 1 leader_uuid: "a878cdea1c924a35a02b84e72f429493" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a878cdea1c924a35a02b84e72f429493" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44349 } } }
I20260619 04:46:46.948210 14421 sys_catalog.cc:458] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:46.948208 14422 sys_catalog.cc:458] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493 [sys.catalog]: This master's current role is: LEADER
I20260619 04:46:46.949298 14426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:46:46.954438 14426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:46:46.964946 14426 catalog_manager.cc:1347] Generated new cluster ID: 43d37941f5964814a6a9c7083ee28ec2
I20260619 04:46:46.965060 14426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:46:47.042850 14426 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:46:47.044309 14426 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:46:47.053978 14426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493: Generated new TSK 0
I20260619 04:46:47.054828 14426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:46:47.121326 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44349
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:47.452629 14441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:47.452924 14441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:47.453028 14441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:47.460619 14441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:47.460729 14441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:47.460847 14441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:46:47.470418 14441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:47.471762 14441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:47.473125 14441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:47.486887 14449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:47.486933 14447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:47.486847 14446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:47.487679 14441 server_base.cc:1034] running on GCE node
I20260619 04:46:47.488405 14441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:47.489346 14441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:47.490603 14441 hybrid_clock.cc:648] HybridClock initialized: now 1781844407490565 us; error 47 us; skew 500 ppm
I20260619 04:46:47.490937 14441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:47 dist-test-slave-2qp8 krb5kdc[14335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:47.494077 14441 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:46:47.495560 14441 webserver.cc:460] Webserver started at http://127.3.243.129:42831/ using document root <none> and password file <none>
I20260619 04:46:47.496315 14441 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:47.496466 14441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:47.496765 14441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:47.499644 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "a1f277c355234cc0b111d48f3acda430"
format_stamp: "Formatted at 2026-06-19 04:46:47 on dist-test-slave-2qp8"
server_key: "6a0f32982d990651d0fcded25518c969"
server_key_iv: "e345013b9b13170cf623e619dd1cf2cb"
server_key_version: "encryptionkey@0"
I20260619 04:46:47.500484 14441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "a1f277c355234cc0b111d48f3acda430"
format_stamp: "Formatted at 2026-06-19 04:46:47 on dist-test-slave-2qp8"
server_key: "6a0f32982d990651d0fcded25518c969"
server_key_iv: "e345013b9b13170cf623e619dd1cf2cb"
server_key_version: "encryptionkey@0"
I20260619 04:46:47.505465 14441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 04:46:47.508911 14456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:47.510228 14441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:46:47.510421 14441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "a1f277c355234cc0b111d48f3acda430"
format_stamp: "Formatted at 2026-06-19 04:46:47 on dist-test-slave-2qp8"
server_key: "6a0f32982d990651d0fcded25518c969"
server_key_iv: "e345013b9b13170cf623e619dd1cf2cb"
server_key_version: "encryptionkey@0"
I20260619 04:46:47.510630 14441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:47.525444 14441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:47.529292 14441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:47.529598 14441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:47.530510 14441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:47.531744 14441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:47.531862 14441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:47.532011 14441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:47.532090 14441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:47.564544 14441 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:38095
I20260619 04:46:47.564584 14568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:38095 every 8 connection(s)
I20260619 04:46:47.565789 14441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:46:47.573472 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 14441
I20260619 04:46:47.573731 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:46:47.574390 4046 external_mini_cluster.cc:1428] Setting key 402518b207b32c7bfad6f4f87f32e343
Jun 19 04:46:47 dist-test-slave-2qp8 krb5kdc[14335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:46:47.585505 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:47.568292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:52455 (local address 127.3.243.190:44349)
0619 04:46:47.568730 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:47.568734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:47.569691 (+ 957us) server_negotiation.cc:407] Connection header received
0619 04:46:47.571444 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:47.571449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:47.571537 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:47.571649 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:47.573168 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:47.574004 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:47.575786 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:47.576332 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:47.576467 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:47.579783 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:47.579788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:47.579791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:47.579821 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:47.582515 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:47.583204 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:47.583211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:47.583214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:47.583336 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:47.583751 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:47.583757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:47.583760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:47.584039 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:47.584249 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:47.584929 (+ 680us) server_negotiation.cc:299] Negotiation successful
0619 04:46:47.585147 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":203,"threads_started":1}
I20260619 04:46:47.586498 14572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:47.568867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44349 (local address 127.3.243.129:52455)
0619 04:46:47.569540 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:47.569580 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:47.571178 (+ 1598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:47.571877 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:47.571885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:47.572416 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:47.572968 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:47.572985 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:47.574164 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:47.574168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:47.575625 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:47.575636 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:47.576552 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:47.576560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:47.576788 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:47.577492 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:47.577513 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:47.579576 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:47.582649 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:47.582658 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:47.582674 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:47.583084 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:47.583450 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:47.583455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:47.583459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:47.583641 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:47.584226 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:47.584233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:47.584579 (+ 346us) client_negotiation.cc:769] Sending connection context
0619 04:46:47.584842 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 04:46:47.585358 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":265,"threads_started":1}
I20260619 04:46:47.588153 14569 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44349
I20260619 04:46:47.588460 14569 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:47.589178 14569 heartbeater.cc:511] Master 127.3.243.190:44349 requested a full tablet report, sending...
I20260619 04:46:47.591542 14380 ts_manager.cc:194] Registered new tserver with Master: a1f277c355234cc0b111d48f3acda430 (127.3.243.129:38095)
I20260619 04:46:47.593926 14380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:52455
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:46:47.627374 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44349
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:47.938388 14575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:47.938670 14575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:47.938759 14575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:47.945837 14575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:47.945945 14575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:47.946132 14575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:46:47.955528 14575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:47.956934 14575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:47.958222 14575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:47.970767 14581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:47.970728 14580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:47.970858 14583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:47.971768 14575 server_base.cc:1034] running on GCE node
I20260619 04:46:47.972381 14575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:47.973225 14575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:47.974499 14575 hybrid_clock.cc:648] HybridClock initialized: now 1781844407974453 us; error 55 us; skew 500 ppm
I20260619 04:46:47.974905 14575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:47 dist-test-slave-2qp8 krb5kdc[14335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:47.978024 14575 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:46:47.979303 14575 webserver.cc:460] Webserver started at http://127.3.243.130:34057/ using document root <none> and password file <none>
I20260619 04:46:47.979918 14575 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:47.980016 14575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:47.980325 14575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:47.983103 14575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "d7a7a360e00f46bd9881962779fc895e"
format_stamp: "Formatted at 2026-06-19 04:46:47 on dist-test-slave-2qp8"
server_key: "fc3f02e807639e36bbcdaf878c5c0169"
server_key_iv: "6e1a3fab48f83769dcb1a4187bdb9516"
server_key_version: "encryptionkey@0"
I20260619 04:46:47.983834 14575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "d7a7a360e00f46bd9881962779fc895e"
format_stamp: "Formatted at 2026-06-19 04:46:47 on dist-test-slave-2qp8"
server_key: "fc3f02e807639e36bbcdaf878c5c0169"
server_key_iv: "6e1a3fab48f83769dcb1a4187bdb9516"
server_key_version: "encryptionkey@0"
I20260619 04:46:47.988953 14575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:46:47.992143 14590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:47.993420 14575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:46:47.993584 14575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "d7a7a360e00f46bd9881962779fc895e"
format_stamp: "Formatted at 2026-06-19 04:46:47 on dist-test-slave-2qp8"
server_key: "fc3f02e807639e36bbcdaf878c5c0169"
server_key_iv: "6e1a3fab48f83769dcb1a4187bdb9516"
server_key_version: "encryptionkey@0"
I20260619 04:46:47.993775 14575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:48.007508 14575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:48.011420 14575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:48.011727 14575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:48.012749 14575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:48.014135 14575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:48.014236 14575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:48.014379 14575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:48.014458 14575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:48.047067 14575 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:43021
I20260619 04:46:48.047077 14702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:43021 every 8 connection(s)
I20260619 04:46:48.048425 14575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:46:48.048915 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 14575
I20260619 04:46:48.049124 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:46:48.049698 4046 external_mini_cluster.cc:1428] Setting key d61528c22d49b41c91e785ada6762b43
Jun 19 04:46:48 dist-test-slave-2qp8 krb5kdc[14335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:46:48.072300 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.051516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:53551 (local address 127.3.243.190:44349)
0619 04:46:48.051708 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.051714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.053074 (+ 1360us) server_negotiation.cc:407] Connection header received
0619 04:46:48.055927 (+ 2853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.055931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.056016 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.056163 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:48.058100 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.058760 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.060958 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.061685 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.061821 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.065313 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:48.065322 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:48.065329 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:48.065373 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:48.069147 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:48.069855 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:48.069864 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:48.069870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:48.070035 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:48.070436 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:48.070444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:48.070450 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:48.070833 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:48.070988 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.071603 (+ 615us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.071863 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:46:48.073100 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.052006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44349 (local address 127.3.243.130:53551)
0619 04:46:48.052892 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:48.052942 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:48.055589 (+ 2647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:48.056401 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:48.056415 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:48.057168 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:48.057936 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.057962 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.058900 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.058907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.060811 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.060822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.061887 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.061891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.062069 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.062734 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:48.062754 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:48.065068 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:48.069296 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:48.069302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:48.069314 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:48.069711 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:48.070138 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:48.070142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:48.070144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:48.070309 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:48.070952 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:48.070958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:48.071237 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 04:46:48.071455 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 04:46:48.071917 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":314,"threads_started":1}
I20260619 04:46:48.074659 14703 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44349
I20260619 04:46:48.074975 14703 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:46:48.075706 14703 heartbeater.cc:511] Master 127.3.243.190:44349 requested a full tablet report, sending...
I20260619 04:46:48.077157 14380 ts_manager.cc:194] Registered new tserver with Master: d7a7a360e00f46bd9881962779fc895e (127.3.243.130:43021)
I20260619 04:46:48.078065 14380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:53551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:46:48.103756 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44349
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:46:48.438171 14708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:46:48.438486 14708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:46:48.438694 14708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:46:48.446377 14708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:46:48.446532 14708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:46:48.446750 14708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:46:48.457139 14708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:46:48.458478 14708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:46:48.459874 14708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:46:48.473152 14714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:48.473178 14713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:46:48.473614 14716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:46:48.474066 14708 server_base.cc:1034] running on GCE node
I20260619 04:46:48.474701 14708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:46:48.475631 14708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:46:48.476862 14708 hybrid_clock.cc:648] HybridClock initialized: now 1781844408476808 us; error 66 us; skew 500 ppm
I20260619 04:46:48.477181 14708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:46:48 dist-test-slave-2qp8 krb5kdc[14335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:46:48.483578 14708 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:46:48.485266 14708 webserver.cc:460] Webserver started at http://127.3.243.131:39303/ using document root <none> and password file <none>
I20260619 04:46:48.485993 14708 fs_manager.cc:359] Metadata directory not provided
I20260619 04:46:48.486174 14708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:46:48.486500 14708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:46:48.489581 14708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "326f805867234302b53de963584dc0dd"
format_stamp: "Formatted at 2026-06-19 04:46:48 on dist-test-slave-2qp8"
server_key: "db75a677d2b0b243fbe326effd58d738"
server_key_iv: "1af850f9264281923cecb8713cb5482a"
server_key_version: "encryptionkey@0"
I20260619 04:46:48.490412 14708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "326f805867234302b53de963584dc0dd"
format_stamp: "Formatted at 2026-06-19 04:46:48 on dist-test-slave-2qp8"
server_key: "db75a677d2b0b243fbe326effd58d738"
server_key_iv: "1af850f9264281923cecb8713cb5482a"
server_key_version: "encryptionkey@0"
I20260619 04:46:48.495841 14708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 04:46:48.499714 14723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:48.501221 14708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:46:48.501415 14708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "326f805867234302b53de963584dc0dd"
format_stamp: "Formatted at 2026-06-19 04:46:48 on dist-test-slave-2qp8"
server_key: "db75a677d2b0b243fbe326effd58d738"
server_key_iv: "1af850f9264281923cecb8713cb5482a"
server_key_version: "encryptionkey@0"
I20260619 04:46:48.501636 14708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:46:48.524575 14708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:46:48.529659 14708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:46:48.529942 14708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:46:48.530898 14708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:46:48.532351 14708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:46:48.532430 14708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:48.532532 14708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:46:48.532586 14708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:46:48.567658 14708 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:46533
I20260619 04:46:48.567749 14835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:46533 every 8 connection(s)
I20260619 04:46:48.569149 14708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:46:48.575798 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 14708
I20260619 04:46:48.576015 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:46:48.576454 4046 external_mini_cluster.cc:1428] Setting key f15f8c5df89a9869d1c90cc5d772fd12
Jun 19 04:46:48 dist-test-slave-2qp8 krb5kdc[14335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:46:48.589195 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.571632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:41897 (local address 127.3.243.190:44349)
0619 04:46:48.571839 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.571845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.573075 (+ 1230us) server_negotiation.cc:407] Connection header received
0619 04:46:48.575016 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.575021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.575090 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.575191 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:48.577059 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.577887 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.580179 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.580578 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.580673 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.584053 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:48.584058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:48.584061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:48.584092 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:48.586445 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:48.587065 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:48.587068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:48.587070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:48.587200 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:48.587575 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:48.587579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:48.587581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:48.587844 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:48.587960 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.588786 (+ 826us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.588953 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:46:48.590341 14838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.572155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44349 (local address 127.3.243.131:41897)
0619 04:46:48.572908 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:48.572957 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:48.574757 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:48.575484 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:48.575492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:48.576175 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:46:48.576840 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.576859 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.578050 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.578054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.579995 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.580012 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.580786 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.580792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.581000 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.581633 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:46:48.581667 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:46:48.583858 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:46:48.586564 (+ 2706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:48.586571 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:48.586582 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:48.586970 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:48.587285 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:46:48.587288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:46:48.587290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:46:48.587480 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:46:48.587992 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:46:48.587999 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:46:48.588442 (+ 443us) client_negotiation.cc:769] Sending connection context
0619 04:46:48.588733 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 04:46:48.589141 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":312,"threads_started":1}
I20260619 04:46:48.592048 14836 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44349
I20260619 04:46:48.592384 14836 heartbeater.cc:464] Registering TS with master...
I20260619 04:46:48.593549 14836 heartbeater.cc:511] Master 127.3.243.190:44349 requested a full tablet report, sending...
I20260619 04:46:48.595363 14380 ts_manager.cc:194] Registered new tserver with Master: 326f805867234302b53de963584dc0dd (127.3.243.131:46533)
I20260619 04:46:48.596657 14380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:41897
I20260619 04:46:48.597211 14569 heartbeater.cc:503] Master 127.3.243.190:44349 was elected leader, sending a full tablet report...
I20260619 04:46:48.604692 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:46:48.628237 14571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.610949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.3.243.190:44349)
0619 04:46:48.612295 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.612300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.612315 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:46:48.612361 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.612369 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.612426 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.612478 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:48.614706 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.615882 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.617461 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.618203 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.618337 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.621596 (+ 3259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:48.621605 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:48.621612 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:48.621652 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:48.625360 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:48.626172 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:48.626178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:48.626182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:48.626340 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:48.626698 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:48.626705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:48.626709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:48.627048 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:48.627293 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.627636 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.627861 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228}
I20260619 04:46:48.632967 14380 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:46:48.643698 14380 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:46:48.675330 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.655778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.3.243.130:43021)
0619 04:46:48.666997 (+ 11219us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.667005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.667037 (+ 32us) server_negotiation.cc:407] Connection header received
0619 04:46:48.667162 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.667173 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.667502 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.667728 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:48.668849 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.670451 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.673042 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.674209 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.674368 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.674568 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.674839 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.675006 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11008,"thread_start_us":10856,"threads_started":1}
I20260619 04:46:48.675367 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.655521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:43021 (local address 127.0.0.1:56302)
0619 04:46:48.664598 (+ 9077us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:48.664642 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:48.664806 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:48.668080 (+ 3274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:48.668092 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:48.668143 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:48.668646 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.668660 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.670586 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.670592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.672823 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.672842 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.674411 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.674417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.674599 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.674653 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 04:46:48.674785 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 04:46:48.674963 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8832,"mutex_wait_us":29,"thread_start_us":12930,"threads_started":1}
I20260619 04:46:48.676371 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.655757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:46533 (local address 127.0.0.1:55158)
0619 04:46:48.667250 (+ 11493us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:48.667276 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:48.667421 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:48.669310 (+ 1889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:48.669315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:48.669353 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:48.669743 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.669755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.671339 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.671347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.673446 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.673461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.675492 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.675498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.675661 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.675693 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:46:48.675779 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 04:46:48.675920 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11365,"mutex_wait_us":4,"thread_start_us":8468,"threads_started":1}
I20260619 04:46:48.676959 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.656607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.3.243.129:38095)
0619 04:46:48.667924 (+ 11317us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.667934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.668865 (+ 931us) server_negotiation.cc:407] Connection header received
0619 04:46:48.669021 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.669026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.669269 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.669447 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:48.670346 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.671863 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.674478 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.675885 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.676025 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.676268 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.676469 (+ 201us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.676636 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11163,"thread_start_us":11275,"threads_started":1}
I20260619 04:46:48.676867 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.656082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:38095 (local address 127.0.0.1:43230)
0619 04:46:48.668757 (+ 12675us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:48.668780 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:48.668887 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:48.669716 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:48.669725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:48.669745 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:48.670175 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.670194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.672004 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.672009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.674284 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.674298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.676083 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.676089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.676273 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.676312 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 04:46:48.676412 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 04:46:48.676525 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12533,"mutex_wait_us":66,"thread_start_us":11241,"threads_started":1}
I20260619 04:46:48.677474 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.656526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.3.243.131:46533)
0619 04:46:48.668492 (+ 11966us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.668500 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.668531 (+ 31us) server_negotiation.cc:407] Connection header received
0619 04:46:48.668623 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.668631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.668857 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.669057 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:48.669988 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.671204 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.673611 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.675035 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.675152 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.675353 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.676690 (+ 1337us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.676857 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11755,"thread_start_us":11564,"threads_started":1}
I20260619 04:46:48.679783 14638 tablet_service.cc:1467] Processing CreateTablet for tablet 4358340e85be4d4ea44cb7cff7da5204 (DEFAULT_TABLE table=test-table [id=660bd611ed224edd92195c2aaa545091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:48.680521 14504 tablet_service.cc:1467] Processing CreateTablet for tablet 4358340e85be4d4ea44cb7cff7da5204 (DEFAULT_TABLE table=test-table [id=660bd611ed224edd92195c2aaa545091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:48.681278 14638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4358340e85be4d4ea44cb7cff7da5204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:48.680809 14771 tablet_service.cc:1467] Processing CreateTablet for tablet 4358340e85be4d4ea44cb7cff7da5204 (DEFAULT_TABLE table=test-table [id=660bd611ed224edd92195c2aaa545091]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:46:48.681700 14504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4358340e85be4d4ea44cb7cff7da5204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:48.682087 14771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4358340e85be4d4ea44cb7cff7da5204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:46:48.693027 14854 tablet_bootstrap.cc:492] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: Bootstrap starting.
I20260619 04:46:48.695909 14855 tablet_bootstrap.cc:492] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: Bootstrap starting.
I20260619 04:46:48.696211 14854 tablet_bootstrap.cc:654] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:48.698562 14854 log.cc:824] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:48.699667 14855 tablet_bootstrap.cc:654] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:48.700902 14855 log.cc:824] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:48.701623 14856 tablet_bootstrap.cc:492] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: Bootstrap starting.
I20260619 04:46:48.701983 14854 tablet_bootstrap.cc:492] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: No bootstrap required, opened a new log
I20260619 04:46:48.702255 14854 ts_tablet_manager.cc:1397] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 04:46:48.703357 14855 tablet_bootstrap.cc:492] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: No bootstrap required, opened a new log
I20260619 04:46:48.703610 14855 ts_tablet_manager.cc:1397] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 04:46:48.705022 14856 tablet_bootstrap.cc:654] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: Neither blocks nor log segments found. Creating new log.
I20260619 04:46:48.706311 14856 log.cc:824] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: Log is configured to *not* fsync() on all Append() calls
I20260619 04:46:48.710741 14856 tablet_bootstrap.cc:492] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: No bootstrap required, opened a new log
I20260619 04:46:48.711011 14856 ts_tablet_manager.cc:1397] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260619 04:46:48.711973 14855 raft_consensus.cc:348] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } }
I20260619 04:46:48.712395 14855 raft_consensus.cc:374] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:48.712549 14855 raft_consensus.cc:729] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7a7a360e00f46bd9881962779fc895e, State: Initialized, Role: FOLLOWER
I20260619 04:46:48.712849 14854 raft_consensus.cc:348] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } }
I20260619 04:46:48.713253 14854 raft_consensus.cc:374] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:48.713356 14854 raft_consensus.cc:729] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f277c355234cc0b111d48f3acda430, State: Initialized, Role: FOLLOWER
I20260619 04:46:48.713171 14855 consensus_queue.cc:260] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } }
I20260619 04:46:48.713924 14854 consensus_queue.cc:260] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } }
I20260619 04:46:48.715744 14854 ts_tablet_manager.cc:1428] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 04:46:48.720609 14860 raft_consensus.cc:482] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:46:48.721410 14703 heartbeater.cc:503] Master 127.3.243.190:44349 was elected leader, sending a full tablet report...
I20260619 04:46:48.722029 14855 ts_tablet_manager.cc:1428] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: Time spent starting tablet: real 0.018s user 0.013s sys 0.005s
I20260619 04:46:48.722858 14856 raft_consensus.cc:348] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } }
I20260619 04:46:48.723145 14860 raft_consensus.cc:504] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } }
I20260619 04:46:48.723500 14856 raft_consensus.cc:374] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:46:48.723614 14856 raft_consensus.cc:729] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326f805867234302b53de963584dc0dd, State: Initialized, Role: FOLLOWER
I20260619 04:46:48.724453 14856 consensus_queue.cc:260] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } }
I20260619 04:46:48.731874 14836 heartbeater.cc:503] Master 127.3.243.190:44349 was elected leader, sending a full tablet report...
I20260619 04:46:48.732271 14860 leader_election.cc:288] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 326f805867234302b53de963584dc0dd (127.3.243.131:46533), a1f277c355234cc0b111d48f3acda430 (127.3.243.129:38095)
I20260619 04:46:48.733085 14856 ts_tablet_manager.cc:1428] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: Time spent starting tablet: real 0.022s user 0.022s sys 0.000s
I20260619 04:46:48.741324 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.726309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:46123 (local address 127.3.243.131:46533)
0619 04:46:48.726548 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.726557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.734488 (+ 7931us) server_negotiation.cc:407] Connection header received
0619 04:46:48.734791 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.734799 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.734885 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.734990 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:48.735591 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.736600 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.739074 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.740302 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.740510 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.740594 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.740958 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.741096 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 04:46:48.741400 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.733581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:37443 (local address 127.3.243.129:38095)
0619 04:46:48.733835 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.733844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.734729 (+ 885us) server_negotiation.cc:407] Connection header received
0619 04:46:48.734976 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.734981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.735057 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.735146 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:46:48.735920 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.736996 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.739275 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.740523 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.740671 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.740718 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.741068 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.741187 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:46:48.741488 14864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.732496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:38095 (local address 127.3.243.130:37443)
0619 04:46:48.734642 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:48.734659 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:48.734816 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:48.735298 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:48.735305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:48.735403 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:48.735765 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.735772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.737138 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.737145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.739134 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.739144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.740705 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.740710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.740922 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.740950 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:46:48.741024 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 04:46:48.741118 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2035,"mutex_wait_us":4,"thread_start_us":1577,"threads_started":1}
I20260619 04:46:48.741726 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.725957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:46533 (local address 127.3.243.130:46123)
0619 04:46:48.734367 (+ 8410us) negotiation.cc:107] Waiting for socket to connect
0619 04:46:48.734391 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:46:48.734608 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:46:48.734993 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:46:48.734998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:46:48.735018 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:46:48.735415 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.735428 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.736726 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.736734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.738918 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:46:48.738932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.740525 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.740529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:46:48.740784 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.740827 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 04:46:48.740928 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 04:46:48.741067 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8231,"mutex_wait_us":42,"thread_start_us":8426,"threads_started":1}
I20260619 04:46:48.742465 14524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4358340e85be4d4ea44cb7cff7da5204" candidate_uuid: "d7a7a360e00f46bd9881962779fc895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f277c355234cc0b111d48f3acda430" is_pre_election: true
I20260619 04:46:48.742630 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4358340e85be4d4ea44cb7cff7da5204" candidate_uuid: "d7a7a360e00f46bd9881962779fc895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326f805867234302b53de963584dc0dd" is_pre_election: true
I20260619 04:46:48.742995 14524 raft_consensus.cc:2454] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7a7a360e00f46bd9881962779fc895e in term 0.
I20260619 04:46:48.743117 14791 raft_consensus.cc:2454] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7a7a360e00f46bd9881962779fc895e in term 0.
I20260619 04:46:48.743677 14592 leader_election.cc:302] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1f277c355234cc0b111d48f3acda430, d7a7a360e00f46bd9881962779fc895e; no voters:
I20260619 04:46:48.744088 14862 raft_consensus.cc:2781] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:46:48.744251 14862 raft_consensus.cc:482] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:46:48.744395 14862 raft_consensus.cc:3037] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:48.746743 14862 raft_consensus.cc:504] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } }
I20260619 04:46:48.747515 14862 leader_election.cc:288] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [CANDIDATE]: Term 1 election: Requested vote from peers 326f805867234302b53de963584dc0dd (127.3.243.131:46533), a1f277c355234cc0b111d48f3acda430 (127.3.243.129:38095)
I20260619 04:46:48.747869 14791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4358340e85be4d4ea44cb7cff7da5204" candidate_uuid: "d7a7a360e00f46bd9881962779fc895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326f805867234302b53de963584dc0dd"
I20260619 04:46:48.747929 14524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4358340e85be4d4ea44cb7cff7da5204" candidate_uuid: "d7a7a360e00f46bd9881962779fc895e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1f277c355234cc0b111d48f3acda430"
I20260619 04:46:48.748042 14791 raft_consensus.cc:3037] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:48.748169 14524 raft_consensus.cc:3037] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:46:48.750193 14791 raft_consensus.cc:2454] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7a7a360e00f46bd9881962779fc895e in term 1.
I20260619 04:46:48.750319 14524 raft_consensus.cc:2454] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7a7a360e00f46bd9881962779fc895e in term 1.
I20260619 04:46:48.750586 14591 leader_election.cc:302] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 326f805867234302b53de963584dc0dd, d7a7a360e00f46bd9881962779fc895e; no voters:
I20260619 04:46:48.750936 14862 raft_consensus.cc:2781] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:46:48.751065 14862 raft_consensus.cc:686] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 1 LEADER]: Becoming Leader. State: Replica: d7a7a360e00f46bd9881962779fc895e, State: Running, Role: LEADER
I20260619 04:46:48.751541 14862 consensus_queue.cc:237] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } }
I20260619 04:46:48.755570 14379 catalog_manager.cc:5495] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e reported cstate change: term changed from 0 to 1, leader changed from <none> to d7a7a360e00f46bd9881962779fc895e (127.3.243.130). New cstate: current_term: 1 leader_uuid: "d7a7a360e00f46bd9881962779fc895e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7a7a360e00f46bd9881962779fc895e" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 43021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 } health_report { overall_health: UNKNOWN } } }
I20260619 04:46:48.778373 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.773416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.3.243.130:43021)
0619 04:46:48.773619 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.773624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.773641 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:46:48.773777 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.773781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.773863 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.773983 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:46:48.774299 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.774985 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.776336 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.776741 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.776832 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.777265 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:46:48.777785 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:46:48.777891 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.777998 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.778062 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 04:46:48.786317 14791 raft_consensus.cc:1261] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [term 1 FOLLOWER]: Refusing update from remote peer d7a7a360e00f46bd9881962779fc895e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:48.786317 14524 raft_consensus.cc:1261] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [term 1 FOLLOWER]: Refusing update from remote peer d7a7a360e00f46bd9881962779fc895e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:46:48.787304 14860 consensus_queue.cc:1035] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1f277c355234cc0b111d48f3acda430" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:48.787606 14862 consensus_queue.cc:1035] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [LEADER]: Connected to new peer: Peer: permanent_uuid: "326f805867234302b53de963584dc0dd" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:46:48.798478 14868 mvcc.cc:205] Tried to move back new op lower bound from 7298434698378485760 to 7298434698250444800. Current Snapshot: MvccSnapshot[applied={T|T < 7298434698378485760}]
I20260619 04:46:48.800972 14870 mvcc.cc:205] Tried to move back new op lower bound from 7298434698378485760 to 7298434698250444800. Current Snapshot: MvccSnapshot[applied={T|T < 7298434698378485760}]
I20260619 04:46:48.813012 14380 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:48.813318 14380 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:46:48.817150 14380 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a878cdea1c924a35a02b84e72f429493: Sending DeleteTablet for 3 replicas of tablet 4358340e85be4d4ea44cb7cff7da5204
I20260619 04:46:48.818293 14638 tablet_service.cc:1514] Processing DeleteTablet for tablet 4358340e85be4d4ea44cb7cff7da5204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:48 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:56302
I20260619 04:46:48.818426 14771 tablet_service.cc:1514] Processing DeleteTablet for tablet 4358340e85be4d4ea44cb7cff7da5204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:48 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:55158
I20260619 04:46:48.819319 14504 tablet_service.cc:1514] Processing DeleteTablet for tablet 4358340e85be4d4ea44cb7cff7da5204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:46:48 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:43230
I20260619 04:46:48.822042 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 14441
I20260619 04:46:48.833123 14880 tablet_replica.cc:330] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: stopping tablet replica
I20260619 04:46:48.833642 14880 raft_consensus.cc:2229] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:48.834231 14880 raft_consensus.cc:2258] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:48.836648 14880 ts_tablet_manager.cc:1905] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:48.845392 14880 ts_tablet_manager.cc:1918] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:48.845587 14880 log.cc:1196] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/4358340e85be4d4ea44cb7cff7da5204
I20260619 04:46:48.846210 14880 ts_tablet_manager.cc:1939] T 4358340e85be4d4ea44cb7cff7da5204 P a1f277c355234cc0b111d48f3acda430: Deleting consensus metadata
I20260619 04:46:48.848344 14365 catalog_manager.cc:4841] TS a1f277c355234cc0b111d48f3acda430 (127.3.243.129:38095): tablet 4358340e85be4d4ea44cb7cff7da5204 (table test-table [id=660bd611ed224edd92195c2aaa545091]) successfully deleted
I20260619 04:46:48.850067 14879 tablet_replica.cc:330] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: stopping tablet replica
I20260619 04:46:48.850606 14879 raft_consensus.cc:2229] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:46:48.851243 14879 raft_consensus.cc:2258] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:48.853518 14879 ts_tablet_manager.cc:1905] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:46:48.854585 14878 tablet_replica.cc:330] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: stopping tablet replica
I20260619 04:46:48.855024 14878 raft_consensus.cc:2229] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:46:48.855693 14878 raft_consensus.cc:2258] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:46:48.858018 14878 ts_tablet_manager.cc:1905] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 04:46:48 dist-test-slave-2qp8 krb5kdc[14335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:46:48.860885 14879 ts_tablet_manager.cc:1918] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:48.861034 14879 log.cc:1196] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/4358340e85be4d4ea44cb7cff7da5204
I20260619 04:46:48.861645 14879 ts_tablet_manager.cc:1939] T 4358340e85be4d4ea44cb7cff7da5204 P 326f805867234302b53de963584dc0dd: Deleting consensus metadata
I20260619 04:46:48.863492 14364 catalog_manager.cc:4841] TS 326f805867234302b53de963584dc0dd (127.3.243.131:46533): tablet 4358340e85be4d4ea44cb7cff7da5204 (table test-table [id=660bd611ed224edd92195c2aaa545091]) successfully deleted
I20260619 04:46:48.864923 14878 ts_tablet_manager.cc:1918] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:46:48.865115 14878 log.cc:1196] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/4358340e85be4d4ea44cb7cff7da5204
I20260619 04:46:48.865615 14878 ts_tablet_manager.cc:1939] T 4358340e85be4d4ea44cb7cff7da5204 P d7a7a360e00f46bd9881962779fc895e: Deleting consensus metadata
I20260619 04:46:48.867355 14367 catalog_manager.cc:4841] TS d7a7a360e00f46bd9881962779fc895e (127.3.243.130:43021): tablet 4358340e85be4d4ea44cb7cff7da5204 (table test-table [id=660bd611ed224edd92195c2aaa545091]) successfully deleted
I20260619 04:46:48.868492 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:48.823815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43234 (local address 127.3.243.129:38095)
0619 04:46:48.824049 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:48.824059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:48.847258 (+ 23199us) server_negotiation.cc:407] Connection header received
0619 04:46:48.847777 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:48.847782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:48.847866 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:48.847939 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:48.849610 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.850673 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.852498 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:48.853100 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:48.853226 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:48.859435 (+ 6209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:48.859454 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:48.859472 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:48.859523 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:48.864798 (+ 5275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:48.865874 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:48.865886 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:48.865892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:48.866060 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:48.866529 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:48.866534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:48.866537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:48.867011 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:48.867317 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:48.867702 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 04:46:48.867879 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:46:48.869138 14564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:48.987309 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 14441
I20260619 04:46:49.005056 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 14575
Jun 19 04:46:49 dist-test-slave-2qp8 krb5kdc[14335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:46:49.022835 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:49.006002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.3.243.130:43021)
0619 04:46:49.006205 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:49.006214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:49.006237 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:46:49.006446 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:49.006453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:49.006533 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:49.006654 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:49.008045 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:49.009110 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:49.010438 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:49.010944 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:49.011094 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:49.015608 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:49.015627 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:49.015648 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:49.015701 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:49.019611 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:49.020662 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:49.020672 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:49.020677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:49.020851 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:49.021204 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:49.021209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:49.021213 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:49.021707 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:49.022058 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:49.022215 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 04:46:49.022444 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:46:49.023568 14698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:49.137925 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 14575
I20260619 04:46:49.156349 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 14708
Jun 19 04:46:49 dist-test-slave-2qp8 krb5kdc[14335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:46:49.172173 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:46:49.157137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.3.243.131:46533)
0619 04:46:49.157310 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 04:46:49.157314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:46:49.157385 (+ 71us) server_negotiation.cc:407] Connection header received
0619 04:46:49.157551 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:46:49.157555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:46:49.157625 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:46:49.157711 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:46:49.159310 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:49.160522 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:49.162368 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:46:49.162759 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:46:49.162858 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:46:49.167043 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:46:49.167056 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:46:49.167072 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:46:49.167107 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:46:49.169967 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:49.170516 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:49.170523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:49.170526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:49.170649 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:46:49.170956 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:46:49.170959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:46:49.170961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:46:49.171296 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:46:49.171508 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 04:46:49.171712 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 04:46:49.171869 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:46:49.172978 14831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:49.261583 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 14708
I20260619 04:46:49.277920 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 14348
I20260619 04:46:49.279085 14410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:46:49.388964 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 14348
2026-06-19T04:46:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4246 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:46:49 dist-test-slave-2qp8 krb5kdc[14888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:46:49 dist-test-slave-2qp8 krb5kdc[14888](info): set up 2 sockets
Jun 19 04:46:49 dist-test-slave-2qp8 krb5kdc[14888](info): commencing operation
krb5kdc: starting...
W20260619 04:46:50.497129 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:46:50 dist-test-slave-2qp8 krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:46:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:46:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/rangeradmin_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/rangeradmin_127.3.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/rangerlookup_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/rangerlookup_127.3.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/HTTP_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/HTTP_127.3.243.148@KRBTEST.COM.keytab.
I20260619 04:46:50.813380 4046 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAbcqnl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/postgres -l logfile start
2026-06-19 04:46:53.316 UTC [14918] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 04:46:53.316 UTC [14918] LOG: listening on IPv4 address "127.3.243.148", port 39139
2026-06-19 04:46:53.319 UTC [14918] LOG: listening on Unix socket "/tmp/.s.PGSQL.39139"
2026-06-19 04:46:53.341 UTC [14921] LOG: database system was shut down at 2026-06-19 04:46:51 UTC
2026-06-19 04:46:53.346 UTC [14918] LOG: database system is ready to accept connections
I20260619 04:46:54.317091 4046 mini_postgres.cc:96] Postgres bound to 39139
2026-06-19 04:46:54.335 UTC [14929] FATAL: database "slave" does not exist
127.3.243.148:39139 - accepting connections
I20260619 04:46:54.336483 4046 mini_ranger.cc:161] Starting Ranger...
I20260619 04:46:54.362493 4046 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 04:46:54.854235 4046 mini_ranger.cc:87] Created ranger Postgres database
I20260619 04:46:54.854372 4046 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-admin
2026-06-19 04:46:55,148 [I] DB FLAVOR :POSTGRES
2026-06-19 04:46:55,148 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 04:46:55,149 [I] Checking connection..
2026-06-19 04:46:55,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 04:46:55,419 [I] Checking connection passed.
2026-06-19 04:46:55,419 [I] --------- Verifying version history table ---------
2026-06-19 04:46:55,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:46:55,646 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 04:46:55,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:46:55,901 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 04:46:55,901 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 04:46:55,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 04:46:56.148 UTC [14993] WARNING: there is no transaction in progress
2026-06-19 04:46:56,154 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 04:46:56,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:46:56,420 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 04:46:56,420 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 04:46:56,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 04:46:56,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 04:46:56,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2qp8.c.gcp-upstream.internal','N') ;"
2026-06-19 04:46:57,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 04:46:57,381 [I] Table x_portal_user does not exist in database ranger
2026-06-19 04:46:57,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 04:46:57,633 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 04:46:57,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 04:46:57,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 04:46:58,149 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 04:46:58,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 04:46:59.085 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.100 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.117 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.132 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.147 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.164 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.226 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.233 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.247 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.257 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.267 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.281 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.290 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.296 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.303 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59.309 UTC [15101] WARNING: there is no transaction in progress
2026-06-19 04:46:59,917 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 04:46:59,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 04:47:00,168 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 04:47:00,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 04:47:00,432 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 04:47:00,433 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 04:47:00,682 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 04:47:00,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 04:47:00,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 04:47:01,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2qp8.c.gcp-upstream.internal';"
2026-06-19 04:47:01,374 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 04:47:01,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 04:47:01,623 [I] DB_PATCHES have already been applied
I20260619 04:47:01.629042 4046 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAbcqnl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 04:47:01.629134 4046 mini_ranger.cc:193] Using host: 127.3.243.148
I20260619 04:47:01.645819 4046 mini_ranger.cc:227] Ranger admin URL: http://127.3.243.148:50679
Jun 19, 2026 4:47:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:47:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:47:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:47:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:47:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:47:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:47:02 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 04:47:02.770475 4046 mini_ranger.cc:160] Time spent starting Ranger: real 8.434s user 0.000s sys 0.041s
Jun 19, 2026 4:47:21 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:47:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 4:47:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:47:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:47:22 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 04:47:23.448308 4046 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/rangerkms_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/rangerkms_127.3.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.243.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.243.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.243.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/HTTP_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.243.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/HTTP_127.3.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 04:47:23 dist-test-slave-2qp8 krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844443, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 04:47:23.631261 4046 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 04:47:23.659160 4046 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 04:47:24.564147 4046 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 04:47:24.564339 4046 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-kms
I20260619 04:47:24.564465 4046 mini_ranger_kms.cc:227] Using postgres at 127.3.243.148:39139
2026-06-19 04:47:24,992 [I] DB FLAVOR :POSTGRES
2026-06-19 04:47:24,993 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 04:47:24,993 [I] Checking connection
2026-06-19 04:47:24,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 04:47:25,269 [I] connection success
2026-06-19 04:47:25,269 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 04:47:25,269 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 04:47:25,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 04:47:25,523 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 04:47:25,524 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 04:47:25,524 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 04:47:25,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:39139/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 04:47:25,857 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 04:47:26.277201 4046 mini_ranger_kms.cc:310] Created kms service
I20260619 04:47:26.492908 4046 mini_ranger_kms.cc:326] Created kudu user
I20260619 04:47:26.536886 4046 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 04:47:26 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844443, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:47:26.902169 4046 mini_ranger_kms.cc:384] Added ranger policy
I20260619 04:47:26.902515 4046 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 04:47:26.902587 4046 mini_ranger_kms.cc:240] Using host: 127.3.243.148
I20260619 04:47:26.925066 4046 mini_ranger_kms.cc:276] Ranger KMS PID: 15281
I20260619 04:47:26.925253 4046 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.243.148:57493
Jun 19, 2026 4:47:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:47:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:47:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:47:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:47:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:47:28 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 04:47:29.300483 4046 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.669s user 0.126s sys 0.321s
I20260619 04:47:29.300756 4046 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 04:47:29.300987 4046 mini_ranger_kms.cc:401] 127.3.243.148:57493/kms/v1/keys
Jun 19 04:47:29 dist-test-slave-2qp8 krb5kdc[14888](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781844449, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 04:47:29 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844449, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.148@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
Jun 19 04:47:30 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844449, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.148@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
Jun 19 04:47:32 dist-test-slave-2qp8 krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:47:32.549296 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45365
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:45365
--ranger_config_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:57493/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:47:32.877400 15332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:47:32.877732 15332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:47:32.877902 15332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:47:32.884840 15332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:47:32.884984 15332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:47:32.885088 15332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:47:32.885191 15332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:47:32.885282 15332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:47:32.894954 15332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.243.148:57493/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:45365
--ranger_config_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:47:32.896224 15332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:47:32.897583 15332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:47:32.909538 15337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:47:32.909580 15338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:47:32.909508 15340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:47:32.910769 15332 server_base.cc:1034] running on GCE node
I20260619 04:47:32.911450 15332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:47:32.912822 15332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:47:32.914009 15332 hybrid_clock.cc:648] HybridClock initialized: now 1781844452913971 us; error 30 us; skew 500 ppm
I20260619 04:47:32.914299 15332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:47:32 dist-test-slave-2qp8 krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844452, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:47:32.917564 15332 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:47:32.918928 15332 webserver.cc:460] Webserver started at http://127.3.243.190:34205/ using document root <none> and password file <none>
I20260619 04:47:32.919507 15332 fs_manager.cc:359] Metadata directory not provided
I20260619 04:47:32.919595 15332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:47:32.919912 15332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:47:32 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844452, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:47:32.976282 15332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec"
format_stamp: "Formatted at 2026-06-19 04:47:32 on dist-test-slave-2qp8"
server_key: "d5e3860bb9ed3804c6a628dbc56a8dab"
server_key_iv: "94a4d1723754d9e1a7f275a7bae9b2b4"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:32.977332 15332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec"
format_stamp: "Formatted at 2026-06-19 04:47:32 on dist-test-slave-2qp8"
server_key: "d5e3860bb9ed3804c6a628dbc56a8dab"
server_key_iv: "94a4d1723754d9e1a7f275a7bae9b2b4"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:32.984231 15332 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260619 04:47:33.004477 15347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:33.006263 15332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 04:47:33.006453 15332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec"
format_stamp: "Formatted at 2026-06-19 04:47:32 on dist-test-slave-2qp8"
server_key: "d5e3860bb9ed3804c6a628dbc56a8dab"
server_key_iv: "94a4d1723754d9e1a7f275a7bae9b2b4"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:33.006712 15332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:47:33.036984 15332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:47:33.040891 15332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:47:33.041131 15332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:47:33.059832 15332 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:45365
I20260619 04:47:33.059865 15398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:45365 every 8 connection(s)
I20260619 04:47:33.061187 15332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:47:33.064808 15399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:47:33.064986 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 15332
I20260619 04:47:33.065189 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:47:33.065811 4046 external_mini_cluster.cc:1428] Setting key ffc9ac2193c7122eec8c02f1ef40a781
I20260619 04:47:33.078975 15399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec: Bootstrap starting.
Jun 19 04:47:33 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:47:33.082777 15399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec: Neither blocks nor log segments found. Creating new log.
I20260619 04:47:33.084048 15399 log.cc:824] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec: Log is configured to *not* fsync() on all Append() calls
I20260619 04:47:33.086619 15399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec: No bootstrap required, opened a new log
I20260619 04:47:33.089359 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:33.067835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.3.243.190:45365)
0619 04:47:33.069095 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:33.069110 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:33.069156 (+ 46us) server_negotiation.cc:407] Connection header received
0619 04:47:33.071654 (+ 2498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:33.071681 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:33.072213 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:33.072641 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:47:33.073698 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:33.075632 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:33.076987 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:33.077969 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:33.078140 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:33.081970 (+ 3830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:47:33.081980 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:47:33.081996 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:47:33.082050 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:47:33.085633 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:33.086284 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:33.086298 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:33.086304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:33.086454 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:33.086848 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:33.086855 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:33.086858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:33.087315 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:47:33.087607 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:33.088064 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 04:47:33.088452 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":825,"threads_started":1}
I20260619 04:47:33.094357 15399 raft_consensus.cc:348] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45365 } }
I20260619 04:47:33.095073 15399 raft_consensus.cc:374] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:47:33.095185 15399 raft_consensus.cc:729] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34ae7ca6c7a4d1a91c06b57dd759fec, State: Initialized, Role: FOLLOWER
I20260619 04:47:33.095753 15399 consensus_queue.cc:260] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45365 } }
I20260619 04:47:33.095986 15399 raft_consensus.cc:388] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:47:33.096150 15399 raft_consensus.cc:482] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:47:33.096328 15399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:47:33.098541 15399 raft_consensus.cc:504] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45365 } }
I20260619 04:47:33.099001 15399 leader_election.cc:302] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d34ae7ca6c7a4d1a91c06b57dd759fec; no voters:
I20260619 04:47:33.099575 15399 leader_election.cc:288] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:47:33.099707 15404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:47:33.100373 15404 raft_consensus.cc:686] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [term 1 LEADER]: Becoming Leader. State: Replica: d34ae7ca6c7a4d1a91c06b57dd759fec, State: Running, Role: LEADER
I20260619 04:47:33.101038 15404 consensus_queue.cc:237] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45365 } }
I20260619 04:47:33.101663 15399 sys_catalog.cc:564] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:47:33.103924 15406 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d34ae7ca6c7a4d1a91c06b57dd759fec. Latest consensus state: current_term: 1 leader_uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45365 } } }
I20260619 04:47:33.104318 15406 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [sys.catalog]: This master's current role is: LEADER
I20260619 04:47:33.103892 15405 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34ae7ca6c7a4d1a91c06b57dd759fec" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45365 } } }
I20260619 04:47:33.105515 15405 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec [sys.catalog]: This master's current role is: LEADER
I20260619 04:47:33.109392 15412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:47:33.114670 15412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:47:33.115396 15399 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 04:47:33.133361 15412 catalog_manager.cc:1347] Generated new cluster ID: f2c3fb1e63094908b79358ef32dc152b
I20260619 04:47:33.133488 15412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:47:33.154294 15412 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:47:33.156755 15412 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:47:33.173517 15412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec: Generated new TSK 0
I20260619 04:47:33.174616 15412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:47:33.895733 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:57493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45365
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:47:34 dist-test-slave-2qp8 krb5kdc[14888](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781844454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 04:47:34.301090 15444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:47:34.301546 15444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:47:34.301743 15444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:47:34.311154 15444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:47:34.311298 15444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:47:34.311504 15444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:47:34.327565 15444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.243.148:57493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:47:34.329398 15444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:47:34.331241 15444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:47:34.358215 15455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:47:34.366418 15444 server_base.cc:1034] running on GCE node
W20260619 04:47:34.366814 15454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:47:34.367157 15458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:47:34.367992 15444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:47:34.369158 15444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:47:34.370430 15444 hybrid_clock.cc:648] HybridClock initialized: now 1781844454370339 us; error 90 us; skew 500 ppm
I20260619 04:47:34.370767 15444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:47:34 dist-test-slave-2qp8 krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:47:34.378494 15444 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:47:34.380504 15444 webserver.cc:460] Webserver started at http://127.3.243.129:40569/ using document root <none> and password file <none>
I20260619 04:47:34.381237 15444 fs_manager.cc:359] Metadata directory not provided
I20260619 04:47:34.381381 15444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:47:34.381703 15444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:47:34 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:47:34.399117 15444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "100d06bd542b4b62bd2283cc243df5fa"
format_stamp: "Formatted at 2026-06-19 04:47:34 on dist-test-slave-2qp8"
server_key: "b7df95bc7404fd1d41b6ea62e09143e7"
server_key_iv: "0f3358d8ea7e078b90ff94d33d256dc7"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:34.400669 15444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "100d06bd542b4b62bd2283cc243df5fa"
format_stamp: "Formatted at 2026-06-19 04:47:34 on dist-test-slave-2qp8"
server_key: "b7df95bc7404fd1d41b6ea62e09143e7"
server_key_iv: "0f3358d8ea7e078b90ff94d33d256dc7"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:34.418385 15444 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.008s sys 0.008s
I20260619 04:47:34.447863 15464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:34.450358 15444 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.007s sys 0.003s
I20260619 04:47:34.450572 15444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "100d06bd542b4b62bd2283cc243df5fa"
format_stamp: "Formatted at 2026-06-19 04:47:34 on dist-test-slave-2qp8"
server_key: "b7df95bc7404fd1d41b6ea62e09143e7"
server_key_iv: "0f3358d8ea7e078b90ff94d33d256dc7"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:34.450953 15444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:47:34.496173 15444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:47:34.501138 15444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:47:34.501570 15444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:47:34.502754 15444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:47:34.504518 15444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:47:34.504595 15444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:34.504741 15444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:47:34.504810 15444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:34.614310 15444 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:42893
I20260619 04:47:34.614336 15576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:42893 every 8 connection(s)
I20260619 04:47:34.616526 15444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:47:34.618168 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 15444
I20260619 04:47:34.618549 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:47:34.619752 4046 external_mini_cluster.cc:1428] Setting key 9df5bf965e2ed7376b9cc048cabb69cd
Jun 19 04:47:34 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:47:34.662516 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:34.628321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:37865 (local address 127.3.243.190:45365)
0619 04:47:34.637749 (+ 9428us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:34.637754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:34.637775 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:47:34.637835 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:34.637839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:34.637922 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:34.638079 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:47:34.640529 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:34.642008 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:34.645910 (+ 3902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:34.646884 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:34.647051 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:34.652864 (+ 5813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:47:34.652875 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:47:34.652885 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:47:34.652942 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:47:34.657723 (+ 4781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:34.658949 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:34.658954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:34.658956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:34.659100 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:34.660179 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:34.660188 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:34.660198 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:34.660614 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:47:34.660899 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:34.661789 (+ 890us) server_negotiation.cc:299] Negotiation successful
0619 04:47:34.662040 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9296,"thread_start_us":9117,"threads_started":1}
I20260619 04:47:34.663759 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:34.627447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45365 (local address 127.3.243.129:37865)
0619 04:47:34.631475 (+ 4028us) negotiation.cc:107] Waiting for socket to connect
0619 04:47:34.631553 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 04:47:34.635019 (+ 3466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:47:34.638414 (+ 3395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:47:34.638431 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:47:34.639233 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:47:34.640265 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:34.640306 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:34.642280 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:34.642290 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:34.645673 (+ 3383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:34.645712 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:34.647500 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:34.647511 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:34.647901 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:34.648870 (+ 969us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:47:34.648912 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:47:34.652530 (+ 3618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:47:34.657989 (+ 5459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:47:34.658003 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:47:34.658026 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:47:34.658739 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:47:34.659255 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:47:34.659264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:47:34.659271 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:47:34.659983 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:47:34.660961 (+ 978us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:47:34.660968 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:47:34.661342 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 04:47:34.661675 (+ 333us) client_negotiation.cc:240] Negotiation successful
0619 04:47:34.662263 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3683,"thread_start_us":3395,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
I20260619 04:47:34.666496 15577 heartbeater.cc:347] Connected to a master server at 127.3.243.190:45365
I20260619 04:47:34.666860 15577 heartbeater.cc:464] Registering TS with master...
I20260619 04:47:34.667824 15577 heartbeater.cc:511] Master 127.3.243.190:45365 requested a full tablet report, sending...
I20260619 04:47:34.674029 15364 ts_manager.cc:194] Registered new tserver with Master: 100d06bd542b4b62bd2283cc243df5fa (127.3.243.129:42893)
I20260619 04:47:34.676369 15364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:37865
Jun 19 04:47:34 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:47:34.712633 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:57493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45365
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:47:35 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844454, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
W20260619 04:47:35.201334 15583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:47:35.201864 15583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:47:35.202081 15583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:47:35.213981 15583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:47:35.214227 15583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:47:35.214476 15583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:47:35.230964 15583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.243.148:57493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:47:35.232976 15583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:47:35.234731 15583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:47:35.270659 15590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:47:35.272734 15591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:47:35.279875 15594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:47:35.280303 15583 server_base.cc:1034] running on GCE node
I20260619 04:47:35.280956 15583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:47:35.282099 15583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:47:35.283386 15583 hybrid_clock.cc:648] HybridClock initialized: now 1781844455283313 us; error 62 us; skew 500 ppm
I20260619 04:47:35.283732 15583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:47:35 dist-test-slave-2qp8 krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:47:35.286940 15583 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:47:35.297348 15583 webserver.cc:460] Webserver started at http://127.3.243.130:35329/ using document root <none> and password file <none>
I20260619 04:47:35.298120 15583 fs_manager.cc:359] Metadata directory not provided
I20260619 04:47:35.298269 15583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:47:35.298609 15583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:47:35 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:47:35.320555 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "ccf9b1b45557499781d00f984294e9ad"
format_stamp: "Formatted at 2026-06-19 04:47:35 on dist-test-slave-2qp8"
server_key: "6723725fe093a8566e7899e15cbd2752"
server_key_iv: "12d66a3b988f1fe1810bbba901f0b0a9"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:35.321468 15583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "ccf9b1b45557499781d00f984294e9ad"
format_stamp: "Formatted at 2026-06-19 04:47:35 on dist-test-slave-2qp8"
server_key: "6723725fe093a8566e7899e15cbd2752"
server_key_iv: "12d66a3b988f1fe1810bbba901f0b0a9"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:35.333520 15583 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.011s sys 0.001s
I20260619 04:47:35.354807 15607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:35.356657 15583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 04:47:35.356854 15583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "ccf9b1b45557499781d00f984294e9ad"
format_stamp: "Formatted at 2026-06-19 04:47:35 on dist-test-slave-2qp8"
server_key: "6723725fe093a8566e7899e15cbd2752"
server_key_iv: "12d66a3b988f1fe1810bbba901f0b0a9"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:35.357133 15583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:47:35.409519 15583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:47:35.413683 15583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:47:35.414407 15583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:47:35.415405 15583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:47:35.416901 15583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:47:35.416971 15583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:35.417109 15583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:47:35.417155 15583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:35.454604 15583 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:33339
I20260619 04:47:35.454622 15719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:33339 every 8 connection(s)
I20260619 04:47:35.455888 15583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:47:35.464999 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 15583
I20260619 04:47:35.465209 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:47:35.465694 4046 external_mini_cluster.cc:1428] Setting key 4d095875cab9827c4452b3cb76970d78
Jun 19 04:47:35 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:47:35.477084 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:35.458658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:49463 (local address 127.3.243.190:45365)
0619 04:47:35.459174 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:35.459181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:35.460210 (+ 1029us) server_negotiation.cc:407] Connection header received
0619 04:47:35.462065 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:35.462071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:35.462165 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:35.462304 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:47:35.464314 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:35.465489 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:35.467389 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:35.467937 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:35.468067 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:35.471190 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:47:35.471195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:47:35.471198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:47:35.471228 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:47:35.473967 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:35.474655 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:35.474665 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:35.474668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:35.474786 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:35.475274 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:35.475279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:35.475282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:35.475560 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:47:35.475711 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:35.476461 (+ 750us) server_negotiation.cc:299] Negotiation successful
0619 04:47:35.476715 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":262,"threads_started":1}
I20260619 04:47:35.478335 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:35.459296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45365 (local address 127.3.243.130:49463)
0619 04:47:35.460026 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 04:47:35.460077 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 04:47:35.461815 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:47:35.462499 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:47:35.462507 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:47:35.463002 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:47:35.464091 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:35.464152 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:35.465652 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:35.465657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:35.467245 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:35.467256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:35.468179 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:35.468186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:35.468411 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:35.469052 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:47:35.469083 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:47:35.470983 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:47:35.474102 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:47:35.474111 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:47:35.474130 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:47:35.474535 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:47:35.474887 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:47:35.474893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:47:35.474896 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:47:35.475164 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:47:35.475680 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:47:35.475688 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:47:35.476033 (+ 345us) client_negotiation.cc:769] Sending connection context
0619 04:47:35.476362 (+ 329us) client_negotiation.cc:240] Negotiation successful
0619 04:47:35.476928 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":306,"threads_started":1}
I20260619 04:47:35.479974 15720 heartbeater.cc:347] Connected to a master server at 127.3.243.190:45365
I20260619 04:47:35.480337 15720 heartbeater.cc:464] Registering TS with master...
I20260619 04:47:35.481146 15720 heartbeater.cc:511] Master 127.3.243.190:45365 requested a full tablet report, sending...
I20260619 04:47:35.482828 15364 ts_manager.cc:194] Registered new tserver with Master: ccf9b1b45557499781d00f984294e9ad (127.3.243.130:33339)
I20260619 04:47:35.484134 15364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:49463
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:47:35.521091 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:57493/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45365
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:47:35.680462 15577 heartbeater.cc:503] Master 127.3.243.190:45365 was elected leader, sending a full tablet report...
W20260619 04:47:35.858603 15726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:47:35.858852 15726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:47:35.859313 15726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:47:35.866479 15726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:47:35.866582 15726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:47:35.866695 15726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:47:35.876405 15726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.243.148:57493/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:45365
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:47:35.877609 15726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:47:35.878794 15726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:47:35.891650 15732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:47:35.891686 15734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:47:35.891744 15731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:47:35.892490 15726 server_base.cc:1034] running on GCE node
I20260619 04:47:35.892982 15726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:47:35.894287 15726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:47:35.895476 15726 hybrid_clock.cc:648] HybridClock initialized: now 1781844455895439 us; error 34 us; skew 500 ppm
I20260619 04:47:35.895773 15726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:47:35 dist-test-slave-2qp8 krb5kdc[14888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:47:35.898872 15726 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:47:35.900264 15726 webserver.cc:460] Webserver started at http://127.3.243.131:35753/ using document root <none> and password file <none>
I20260619 04:47:35.900906 15726 fs_manager.cc:359] Metadata directory not provided
I20260619 04:47:35.901031 15726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:47:35.901348 15726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:47:35 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:47:35.918228 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "e1817e389f7f4d87af64c3b553e224ff"
format_stamp: "Formatted at 2026-06-19 04:47:35 on dist-test-slave-2qp8"
server_key: "2c7b4af00b582ab69b692deaa2307753"
server_key_iv: "957735319141dc6513d6dd8b6b070d16"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:35.919335 15726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "e1817e389f7f4d87af64c3b553e224ff"
format_stamp: "Formatted at 2026-06-19 04:47:35 on dist-test-slave-2qp8"
server_key: "2c7b4af00b582ab69b692deaa2307753"
server_key_iv: "957735319141dc6513d6dd8b6b070d16"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:35.924739 15726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 04:47:35.938509 15741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:35.939865 15726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:47:35.940001 15726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "e1817e389f7f4d87af64c3b553e224ff"
format_stamp: "Formatted at 2026-06-19 04:47:35 on dist-test-slave-2qp8"
server_key: "2c7b4af00b582ab69b692deaa2307753"
server_key_iv: "957735319141dc6513d6dd8b6b070d16"
server_key_version: "kuduclusterkey@0"
I20260619 04:47:35.940249 15726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:47:35.968933 15726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:47:35.973204 15726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:47:35.973487 15726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:47:35.974368 15726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:47:35.975718 15726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:47:35.975793 15726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:35.975888 15726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:47:35.975940 15726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:47:36.012527 15726 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:40581
I20260619 04:47:36.012544 15853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:40581 every 8 connection(s)
I20260619 04:47:36.013957 15726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:47:36.023867 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 15726
I20260619 04:47:36.024081 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:47:36.024711 4046 external_mini_cluster.cc:1428] Setting key 065160da2172009cb14307c0881a5d79
Jun 19 04:47:36 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844455, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:47:36.033252 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.016446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:51285 (local address 127.3.243.190:45365)
0619 04:47:36.016898 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.016904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.017895 (+ 991us) server_negotiation.cc:407] Connection header received
0619 04:47:36.019578 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.019582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.019660 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.019755 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:47:36.021187 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.021905 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.023449 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.023855 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.023991 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.027877 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:47:36.027882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:47:36.027885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:47:36.027915 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:47:36.030313 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.031007 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.031014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.031017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.031138 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.031526 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.031533 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.031536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.031800 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:47:36.031966 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.032656 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.032895 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":200,"threads_started":1}
I20260619 04:47:36.034161 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.017020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45365 (local address 127.3.243.131:51285)
0619 04:47:36.017725 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 04:47:36.017762 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:47:36.019336 (+ 1574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:47:36.019957 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:47:36.019965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:47:36.020475 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:47:36.021036 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.021053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.022059 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.022063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.023325 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.023337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.024090 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.024097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.024401 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.025053 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:47:36.025073 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:47:36.027674 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:47:36.030477 (+ 2803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:47:36.030485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:47:36.030502 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:47:36.030895 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:47:36.031241 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:47:36.031247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:47:36.031251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:47:36.031424 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:47:36.031938 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:47:36.031946 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:47:36.032319 (+ 373us) client_negotiation.cc:769] Sending connection context
0619 04:47:36.032591 (+ 272us) client_negotiation.cc:240] Negotiation successful
0619 04:47:36.033099 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":290,"threads_started":1}
I20260619 04:47:36.035378 15854 heartbeater.cc:347] Connected to a master server at 127.3.243.190:45365
I20260619 04:47:36.035630 15854 heartbeater.cc:464] Registering TS with master...
I20260619 04:47:36.036317 15854 heartbeater.cc:511] Master 127.3.243.190:45365 requested a full tablet report, sending...
I20260619 04:47:36.037587 15364 ts_manager.cc:194] Registered new tserver with Master: e1817e389f7f4d87af64c3b553e224ff (127.3.243.131:40581)
I20260619 04:47:36.038666 15364 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:51285
I20260619 04:47:36.040042 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:47:36.057190 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.044623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.3.243.190:45365)
0619 04:47:36.044760 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.044764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.045297 (+ 533us) server_negotiation.cc:407] Connection header received
0619 04:47:36.045454 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.045457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.045525 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.045604 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:47:36.046970 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.047651 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.049019 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.049430 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.049523 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.052069 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:47:36.052074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:47:36.052077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:47:36.052145 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:47:36.055013 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.055590 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.055594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.055596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.055705 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.055995 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.055998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.056000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.056313 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:47:36.056484 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.056715 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.056879 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 04:47:36.060923 15364 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:47:36.069003 15364 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:47:36.107421 15868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.087247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:33339 (local address 127.0.0.1:42904)
0619 04:47:36.088435 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0619 04:47:36.088448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:47:36.088526 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:47:36.099779 (+ 11253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:47:36.099803 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:47:36.099835 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:47:36.100329 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.100346 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.102552 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.102560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.104943 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.104953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.106358 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.106367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.106572 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.106625 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 04:47:36.106767 (+ 142us) client_negotiation.cc:240] Negotiation successful
0619 04:47:36.106961 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"mutex_wait_us":58,"thread_start_us":980,"threads_started":1}
I20260619 04:47:36.107828 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.087819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.3.243.130:33339)
0619 04:47:36.098917 (+ 11098us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.098929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.098962 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:47:36.099076 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.099085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.099350 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.100578 (+ 1228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:47:36.100783 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.102380 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.105108 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.106055 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.106196 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.106423 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.106843 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.107016 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10946,"thread_start_us":10842,"threads_started":1}
I20260619 04:47:36.108111 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.087136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:40581 (local address 127.0.0.1:33244)
0619 04:47:36.087841 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 04:47:36.087880 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:47:36.088034 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:47:36.100156 (+ 12122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:47:36.100162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:47:36.100185 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:47:36.100590 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.100600 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.102496 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.102504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.105127 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.105140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.107167 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.107173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.107362 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.107410 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 04:47:36.107488 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 04:47:36.107640 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":40,"thread_start_us":353,"threads_started":1}
I20260619 04:47:36.108732 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.087390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.3.243.131:40581)
0619 04:47:36.099273 (+ 11883us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.099281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.099322 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:47:36.099410 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.099417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.099657 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.099853 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:47:36.100777 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.102332 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.105321 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.106715 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.106825 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.107024 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.108244 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.108417 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11722,"thread_start_us":12092,"threads_started":1}
I20260619 04:47:36.111076 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.090156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.3.243.129:42893)
0619 04:47:36.102237 (+ 12081us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.102248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.102604 (+ 356us) server_negotiation.cc:407] Connection header received
0619 04:47:36.102850 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.102859 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.103146 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.103324 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:47:36.104355 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.105920 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.108974 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.110009 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.110165 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.110379 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.110605 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.110759 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11903,"thread_start_us":11747,"threads_started":1}
I20260619 04:47:36.111469 15655 tablet_service.cc:1467] Processing CreateTablet for tablet c615e0e83b5443548d9919a63549857b (DEFAULT_TABLE table=test-table [id=1cd077f9f2544d3eb0ac0db75e7ca6db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:47:36.110994 15871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.089901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:42893 (local address 127.0.0.1:51926)
0619 04:47:36.102452 (+ 12551us) negotiation.cc:107] Waiting for socket to connect
0619 04:47:36.102504 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:47:36.102630 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:47:36.103574 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:47:36.103579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:47:36.103606 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:47:36.104146 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.104161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.106067 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.106071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.108775 (+ 2704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.108791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.110201 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.110207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.110413 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.110454 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 04:47:36.110549 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 04:47:36.110667 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12435,"thread_start_us":12361,"threads_started":1}
I20260619 04:47:36.112928 15655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c615e0e83b5443548d9919a63549857b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:47:36.115294 15512 tablet_service.cc:1467] Processing CreateTablet for tablet c615e0e83b5443548d9919a63549857b (DEFAULT_TABLE table=test-table [id=1cd077f9f2544d3eb0ac0db75e7ca6db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:47:36.115847 15789 tablet_service.cc:1467] Processing CreateTablet for tablet c615e0e83b5443548d9919a63549857b (DEFAULT_TABLE table=test-table [id=1cd077f9f2544d3eb0ac0db75e7ca6db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:47:36.116523 15512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c615e0e83b5443548d9919a63549857b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:47:36.117174 15789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c615e0e83b5443548d9919a63549857b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:47:36.129136 15873 tablet_bootstrap.cc:492] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: Bootstrap starting.
I20260619 04:47:36.131752 15874 tablet_bootstrap.cc:492] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: Bootstrap starting.
I20260619 04:47:36.132750 15873 tablet_bootstrap.cc:654] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: Neither blocks nor log segments found. Creating new log.
I20260619 04:47:36.134086 15873 log.cc:824] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: Log is configured to *not* fsync() on all Append() calls
I20260619 04:47:36.135308 15874 tablet_bootstrap.cc:654] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: Neither blocks nor log segments found. Creating new log.
I20260619 04:47:36.135524 15875 tablet_bootstrap.cc:492] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: Bootstrap starting.
I20260619 04:47:36.136566 15874 log.cc:824] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: Log is configured to *not* fsync() on all Append() calls
I20260619 04:47:36.137079 15873 tablet_bootstrap.cc:492] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: No bootstrap required, opened a new log
I20260619 04:47:36.137336 15873 ts_tablet_manager.cc:1397] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 04:47:36.139755 15875 tablet_bootstrap.cc:654] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: Neither blocks nor log segments found. Creating new log.
I20260619 04:47:36.141249 15875 log.cc:824] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: Log is configured to *not* fsync() on all Append() calls
I20260619 04:47:36.144212 15875 tablet_bootstrap.cc:492] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: No bootstrap required, opened a new log
I20260619 04:47:36.144482 15875 ts_tablet_manager.cc:1397] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260619 04:47:36.145427 15873 raft_consensus.cc:348] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } }
I20260619 04:47:36.145805 15873 raft_consensus.cc:374] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:47:36.145912 15873 raft_consensus.cc:729] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100d06bd542b4b62bd2283cc243df5fa, State: Initialized, Role: FOLLOWER
I20260619 04:47:36.145990 15874 tablet_bootstrap.cc:492] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: No bootstrap required, opened a new log
I20260619 04:47:36.146296 15874 ts_tablet_manager.cc:1397] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260619 04:47:36.146912 15873 consensus_queue.cc:260] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } }
I20260619 04:47:36.148594 15873 ts_tablet_manager.cc:1428] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260619 04:47:36.156061 15875 raft_consensus.cc:348] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } }
I20260619 04:47:36.156517 15875 raft_consensus.cc:374] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:47:36.156616 15875 raft_consensus.cc:729] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1817e389f7f4d87af64c3b553e224ff, State: Initialized, Role: FOLLOWER
I20260619 04:47:36.157224 15875 consensus_queue.cc:260] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } }
I20260619 04:47:36.157616 15874 raft_consensus.cc:348] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } }
I20260619 04:47:36.158026 15874 raft_consensus.cc:374] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:47:36.158167 15874 raft_consensus.cc:729] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf9b1b45557499781d00f984294e9ad, State: Initialized, Role: FOLLOWER
I20260619 04:47:36.158236 15854 heartbeater.cc:503] Master 127.3.243.190:45365 was elected leader, sending a full tablet report...
I20260619 04:47:36.158720 15874 consensus_queue.cc:260] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } }
I20260619 04:47:36.158982 15875 ts_tablet_manager.cc:1428] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 04:47:36.159783 15720 heartbeater.cc:503] Master 127.3.243.190:45365 was elected leader, sending a full tablet report...
I20260619 04:47:36.160365 15874 ts_tablet_manager.cc:1428] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260619 04:47:36.208200 15721 tablet.cc:2369] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:47:36.216238 15881 raft_consensus.cc:482] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:47:36.216564 15881 raft_consensus.cc:504] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } }
I20260619 04:47:36.218245 15881 leader_election.cc:288] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1817e389f7f4d87af64c3b553e224ff (127.3.243.131:40581), 100d06bd542b4b62bd2283cc243df5fa (127.3.243.129:42893)
I20260619 04:47:36.225586 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.218554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:43315 (local address 127.3.243.131:40581)
0619 04:47:36.218788 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.218794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.219180 (+ 386us) server_negotiation.cc:407] Connection header received
0619 04:47:36.219384 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.219393 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.219484 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.219579 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:47:36.220267 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.221227 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.223685 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.224635 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.224753 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.224803 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.225185 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.225321 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:47:36.225651 15882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.218438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:40581 (local address 127.3.243.130:43315)
0619 04:47:36.219044 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0619 04:47:36.219067 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:47:36.219226 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:47:36.219638 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:47:36.219646 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:47:36.219683 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:47:36.220093 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.220135 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.221350 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.221356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.223551 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.223562 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.224832 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.224838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.225011 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.225049 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 04:47:36.225155 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 04:47:36.225304 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":63,"thread_start_us":298,"threads_started":1}
I20260619 04:47:36.226161 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.219423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:49921 (local address 127.3.243.129:42893)
0619 04:47:36.219624 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.219632 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.220047 (+ 415us) server_negotiation.cc:407] Connection header received
0619 04:47:36.220227 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.220232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.220306 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.220373 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:47:36.220940 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.221851 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.224170 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.225121 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.225259 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.225321 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.225692 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.225837 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:47:36.226325 15883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.219072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:42893 (local address 127.3.243.130:49921)
0619 04:47:36.219949 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0619 04:47:36.219962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:47:36.220066 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:47:36.220405 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:47:36.220409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:47:36.220432 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:47:36.220775 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.220786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.221974 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.221978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.223977 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:47:36.223991 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.225320 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.225326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:47:36.225514 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.225544 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:47:36.225656 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 04:47:36.225807 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":779,"mutex_wait_us":58,"thread_start_us":590,"threads_started":1}
I20260619 04:47:36.226898 15809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c615e0e83b5443548d9919a63549857b" candidate_uuid: "ccf9b1b45557499781d00f984294e9ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1817e389f7f4d87af64c3b553e224ff" is_pre_election: true
I20260619 04:47:36.227324 15532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c615e0e83b5443548d9919a63549857b" candidate_uuid: "ccf9b1b45557499781d00f984294e9ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100d06bd542b4b62bd2283cc243df5fa" is_pre_election: true
I20260619 04:47:36.227478 15809 raft_consensus.cc:2454] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf9b1b45557499781d00f984294e9ad in term 0.
I20260619 04:47:36.227869 15532 raft_consensus.cc:2454] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccf9b1b45557499781d00f984294e9ad in term 0.
I20260619 04:47:36.228195 15610 leader_election.cc:302] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccf9b1b45557499781d00f984294e9ad, e1817e389f7f4d87af64c3b553e224ff; no voters:
I20260619 04:47:36.228556 15881 raft_consensus.cc:2781] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:47:36.228700 15881 raft_consensus.cc:482] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:47:36.228822 15881 raft_consensus.cc:3037] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:47:36.230876 15881 raft_consensus.cc:504] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } }
I20260619 04:47:36.231487 15881 leader_election.cc:288] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [CANDIDATE]: Term 1 election: Requested vote from peers e1817e389f7f4d87af64c3b553e224ff (127.3.243.131:40581), 100d06bd542b4b62bd2283cc243df5fa (127.3.243.129:42893)
I20260619 04:47:36.231851 15809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c615e0e83b5443548d9919a63549857b" candidate_uuid: "ccf9b1b45557499781d00f984294e9ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1817e389f7f4d87af64c3b553e224ff"
I20260619 04:47:36.231916 15532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c615e0e83b5443548d9919a63549857b" candidate_uuid: "ccf9b1b45557499781d00f984294e9ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100d06bd542b4b62bd2283cc243df5fa"
I20260619 04:47:36.232051 15809 raft_consensus.cc:3037] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:47:36.232086 15532 raft_consensus.cc:3037] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:47:36.235600 15532 raft_consensus.cc:2454] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf9b1b45557499781d00f984294e9ad in term 1.
I20260619 04:47:36.235600 15809 raft_consensus.cc:2454] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccf9b1b45557499781d00f984294e9ad in term 1.
I20260619 04:47:36.235947 15610 leader_election.cc:302] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ccf9b1b45557499781d00f984294e9ad, e1817e389f7f4d87af64c3b553e224ff; no voters:
I20260619 04:47:36.236227 15881 raft_consensus.cc:2781] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:47:36.236634 15881 raft_consensus.cc:686] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 1 LEADER]: Becoming Leader. State: Replica: ccf9b1b45557499781d00f984294e9ad, State: Running, Role: LEADER
I20260619 04:47:36.237082 15881 consensus_queue.cc:237] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } }
I20260619 04:47:36.241530 15364 catalog_manager.cc:5495] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ccf9b1b45557499781d00f984294e9ad (127.3.243.130). New cstate: current_term: 1 leader_uuid: "ccf9b1b45557499781d00f984294e9ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccf9b1b45557499781d00f984294e9ad" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 } health_report { overall_health: UNKNOWN } } }
W20260619 04:47:36.265592 15855 tablet.cc:2369] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:47:36.289873 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.284590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.3.243.130:33339)
0619 04:47:36.284802 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.284808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.284828 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:47:36.285009 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.285013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.285097 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.285244 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:47:36.285603 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.286327 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.287763 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.288199 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.288334 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.288835 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:47:36.289319 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:47:36.289481 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.289532 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.289607 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:47:36.298338 15809 raft_consensus.cc:1261] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [term 1 FOLLOWER]: Refusing update from remote peer ccf9b1b45557499781d00f984294e9ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:47:36.298373 15532 raft_consensus.cc:1261] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [term 1 FOLLOWER]: Refusing update from remote peer ccf9b1b45557499781d00f984294e9ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:47:36.299326 15884 consensus_queue.cc:1035] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1817e389f7f4d87af64c3b553e224ff" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:47:36.299695 15881 consensus_queue.cc:1035] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "100d06bd542b4b62bd2283cc243df5fa" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 42893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:47:36.310207 15888 mvcc.cc:205] Tried to move back new op lower bound from 7298434892987129856 to 7298434892750786560. Current Snapshot: MvccSnapshot[applied={T|T < 7298434892987129856}]
I20260619 04:47:36.329330 15363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:47:36.329603 15363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:47:36.332933 15363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d34ae7ca6c7a4d1a91c06b57dd759fec: Sending DeleteTablet for 3 replicas of tablet c615e0e83b5443548d9919a63549857b
I20260619 04:47:36.334017 15789 tablet_service.cc:1514] Processing DeleteTablet for tablet c615e0e83b5443548d9919a63549857b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:47:36 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:33244
I20260619 04:47:36.334136 15655 tablet_service.cc:1514] Processing DeleteTablet for tablet c615e0e83b5443548d9919a63549857b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:47:36 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:42904
I20260619 04:47:36.335157 15512 tablet_service.cc:1514] Processing DeleteTablet for tablet c615e0e83b5443548d9919a63549857b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:47:36 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:51926
I20260619 04:47:36.336069 15899 tablet_replica.cc:330] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: stopping tablet replica
I20260619 04:47:36.336536 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 15444
I20260619 04:47:36.336544 15899 raft_consensus.cc:2229] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:47:36.337031 15899 raft_consensus.cc:2258] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:47:36.340704 15898 tablet_replica.cc:330] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: stopping tablet replica
I20260619 04:47:36.341194 15898 raft_consensus.cc:2229] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:47:36.341881 15898 raft_consensus.cc:2258] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:47:36.344012 15898 ts_tablet_manager.cc:1905] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:47:36.345431 15899 ts_tablet_manager.cc:1905] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:47:36.351310 15898 ts_tablet_manager.cc:1918] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:47:36.351538 15898 log.cc:1196] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/c615e0e83b5443548d9919a63549857b
I20260619 04:47:36.352159 15898 ts_tablet_manager.cc:1939] T c615e0e83b5443548d9919a63549857b P ccf9b1b45557499781d00f984294e9ad: Deleting consensus metadata
I20260619 04:47:36.354367 15351 catalog_manager.cc:4841] TS ccf9b1b45557499781d00f984294e9ad (127.3.243.130:33339): tablet c615e0e83b5443548d9919a63549857b (table test-table [id=1cd077f9f2544d3eb0ac0db75e7ca6db]) successfully deleted
I20260619 04:47:36.354981 15899 ts_tablet_manager.cc:1918] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:47:36.355194 15899 log.cc:1196] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/c615e0e83b5443548d9919a63549857b
I20260619 04:47:36.355819 15899 ts_tablet_manager.cc:1939] T c615e0e83b5443548d9919a63549857b P 100d06bd542b4b62bd2283cc243df5fa: Deleting consensus metadata
I20260619 04:47:36.360631 15897 tablet_replica.cc:330] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: stopping tablet replica
I20260619 04:47:36.361119 15897 raft_consensus.cc:2229] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:47:36.361668 15897 raft_consensus.cc:2258] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:47:36.362021 15350 catalog_manager.cc:4841] TS 100d06bd542b4b62bd2283cc243df5fa (127.3.243.129:42893): tablet c615e0e83b5443548d9919a63549857b (table test-table [id=1cd077f9f2544d3eb0ac0db75e7ca6db]) successfully deleted
I20260619 04:47:36.364162 15897 ts_tablet_manager.cc:1905] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 04:47:36 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:47:36.370895 15897 ts_tablet_manager.cc:1918] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:47:36.371047 15897 log.cc:1196] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/c615e0e83b5443548d9919a63549857b
I20260619 04:47:36.371526 15897 ts_tablet_manager.cc:1939] T c615e0e83b5443548d9919a63549857b P e1817e389f7f4d87af64c3b553e224ff: Deleting consensus metadata
I20260619 04:47:36.373415 15350 catalog_manager.cc:4841] TS e1817e389f7f4d87af64c3b553e224ff (127.3.243.131:40581): tablet c615e0e83b5443548d9919a63549857b (table test-table [id=1cd077f9f2544d3eb0ac0db75e7ca6db]) successfully deleted
I20260619 04:47:36.375456 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.337856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51928 (local address 127.3.243.129:42893)
0619 04:47:36.338048 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.338054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.358695 (+ 20641us) server_negotiation.cc:407] Connection header received
0619 04:47:36.358852 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.358858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.358960 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.359075 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:47:36.361059 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.362134 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.363773 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.364216 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.364355 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.369053 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:47:36.369065 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:47:36.369079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:47:36.369110 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:47:36.372028 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.372753 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.372760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.372768 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.372891 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.373350 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.373357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.373360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.373815 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:47:36.374127 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.374418 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.374656 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:47:36.376187 15572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:47:36.529821 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 15444
I20260619 04:47:36.547772 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 15583
Jun 19 04:47:36 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:47:36.565107 15870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.548659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.3.243.130:33339)
0619 04:47:36.548858 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.548866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.548916 (+ 50us) server_negotiation.cc:407] Connection header received
0619 04:47:36.549112 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.549115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.549180 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.549291 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:47:36.550882 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.551723 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.553637 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.554047 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.554149 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.558984 (+ 4835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:47:36.558996 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:47:36.559011 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:47:36.559051 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:47:36.562210 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.562956 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.562970 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.562972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.563107 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.563495 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.563501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.563504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.563852 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:47:36.564210 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.564522 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.564737 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:47:36.565805 15715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:47:36.677066 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 15583
I20260619 04:47:36.695688 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 15726
Jun 19 04:47:36 dist-test-slave-2qp8 krb5kdc[14888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:47:36.710582 15869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:47:36.696442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.3.243.131:40581)
0619 04:47:36.696627 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:47:36.696633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:47:36.696738 (+ 105us) server_negotiation.cc:407] Connection header received
0619 04:47:36.696937 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:47:36.696942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:47:36.697019 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:47:36.697124 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:47:36.698413 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.699136 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.700566 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:47:36.700944 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:47:36.701075 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:47:36.705344 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:47:36.705363 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:47:36.705379 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:47:36.705409 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:47:36.707813 (+ 2404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.708527 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.708536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.708539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.708660 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:47:36.709052 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:47:36.709057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:47:36.709060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:47:36.709396 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:47:36.709678 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 04:47:36.710045 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 04:47:36.710247 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:47:36.711158 15849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:47:36.798738 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 15726
I20260619 04:47:36.817222 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 15332
I20260619 04:47:36.818334 15394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:47:36.928241 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 15332
I20260619 04:47:36.948868 4046 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 04:47:37.283564 4046 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 04:47:37.283812 4046 mini_ranger.cc:66] Stopping Ranger...
I20260619 04:47:37.629212 4046 mini_ranger.cc:68] Stopped Ranger
2026-06-19 04:47:37.629 UTC [14918] LOG: received smart shutdown request
2026-06-19 04:47:37.632 UTC [14918] LOG: background worker "logical replication launcher" (PID 14927) exited with exit code 1
2026-06-19 04:47:37.633 UTC [14922] LOG: shutting down
2026-06-19 04:47:37.788 UTC [14918] LOG: database system is shut down
2026-06-19T04:47:37Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48518 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:47:38 dist-test-slave-2qp8 krb5kdc[15913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:47:38 dist-test-slave-2qp8 krb5kdc[15913](info): set up 2 sockets
Jun 19 04:47:38 dist-test-slave-2qp8 krb5kdc[15913](info): commencing operation
krb5kdc: starting...
W20260619 04:47:39.021898 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:47:39 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:47:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:47:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/rangeradmin_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/rangeradmin_127.3.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/rangerlookup_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/rangerlookup_127.3.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/HTTP_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/HTTP_127.3.243.148@KRBTEST.COM.keytab.
I20260619 04:47:39.347055 4046 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAbcqnl/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/postgres -l logfile start
2026-06-19 04:47:41.831 UTC [15943] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 04:47:41.831 UTC [15943] LOG: listening on IPv4 address "127.3.243.148", port 52807
2026-06-19 04:47:41.835 UTC [15943] LOG: listening on Unix socket "/tmp/.s.PGSQL.52807"
2026-06-19 04:47:41.855 UTC [15946] LOG: database system was shut down at 2026-06-19 04:47:40 UTC
2026-06-19 04:47:41.858 UTC [15943] LOG: database system is ready to accept connections
I20260619 04:47:42.836340 4046 mini_postgres.cc:96] Postgres bound to 52807
2026-06-19 04:47:42.858 UTC [15954] FATAL: database "slave" does not exist
127.3.243.148:52807 - accepting connections
I20260619 04:47:42.858788 4046 mini_ranger.cc:161] Starting Ranger...
I20260619 04:47:42.884519 4046 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 04:47:43.367376 4046 mini_ranger.cc:87] Created ranger Postgres database
I20260619 04:47:43.367501 4046 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-admin
2026-06-19 04:47:43,622 [I] DB FLAVOR :POSTGRES
2026-06-19 04:47:43,623 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 04:47:43,623 [I] Checking connection..
2026-06-19 04:47:43,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 04:47:43,848 [I] Checking connection passed.
2026-06-19 04:47:43,848 [I] --------- Verifying version history table ---------
2026-06-19 04:47:43,848 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:47:44,098 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 04:47:44,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:47:44,337 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 04:47:44,337 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 04:47:44,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 04:47:44.598 UTC [16018] WARNING: there is no transaction in progress
2026-06-19 04:47:44,604 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 04:47:44,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 04:47:44,863 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 04:47:44,863 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 04:47:44,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 04:47:45,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 04:47:45,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2qp8.c.gcp-upstream.internal','N') ;"
2026-06-19 04:47:45,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 04:47:45,858 [I] Table x_portal_user does not exist in database ranger
2026-06-19 04:47:45,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 04:47:46,114 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 04:47:46,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 04:47:46,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 04:47:46,598 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 04:47:46,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 04:47:47.551 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.566 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.581 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.597 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.613 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.630 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.683 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.690 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.704 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.714 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.724 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.739 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.748 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.755 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.761 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:47.768 UTC [16126] WARNING: there is no transaction in progress
2026-06-19 04:47:48,387 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 04:47:48,387 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 04:47:48,645 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 04:47:48,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 04:47:48,879 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 04:47:48,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 04:47:49,138 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 04:47:49,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 04:47:49,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 04:47:49,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2qp8.c.gcp-upstream.internal';"
2026-06-19 04:47:49,868 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 04:47:49,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 04:47:50,094 [I] DB_PATCHES have already been applied
I20260619 04:47:50.100715 4046 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAbcqnl/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 04:47:50.100821 4046 mini_ranger.cc:193] Using host: 127.3.243.148
I20260619 04:47:50.115687 4046 mini_ranger.cc:227] Ranger admin URL: http://127.3.243.148:52217
Jun 19, 2026 4:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 4:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 4:47:50 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 4:47:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 4:47:51 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 04:47:51.237903 4046 mini_ranger.cc:160] Time spent starting Ranger: real 8.379s user 0.000s sys 0.039s
Jun 19, 2026 4:48:10 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 4:48:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 4:48:10 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 4:48:10 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 4:48:11 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 04:48:12.249632 4046 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/rangerkms_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/rangerkms_127.3.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.243.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.243.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.243.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/HTTP_127.3.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.243.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/HTTP_127.3.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 04:48:12 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844492, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 04:48:12.430145 4046 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 04:48:12.451751 4046 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 04:48:13.326020 4046 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 04:48:13.326113 4046 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-kms
I20260619 04:48:13.326145 4046 mini_ranger_kms.cc:227] Using postgres at 127.3.243.148:52807
2026-06-19 04:48:13,693 [I] DB FLAVOR :POSTGRES
2026-06-19 04:48:13,694 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 04:48:13,694 [I] Checking connection
2026-06-19 04:48:13,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 04:48:13,954 [I] connection success
2026-06-19 04:48:13,954 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 04:48:13,954 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 04:48:13,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 04:48:14,193 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 04:48:14,193 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 04:48:14,194 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 04:48:14,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.243.148:52807/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 04:48:14,449 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 04:48:14.914499 4046 mini_ranger_kms.cc:310] Created kms service
I20260619 04:48:15.087113 4046 mini_ranger_kms.cc:326] Created kudu user
I20260619 04:48:15.133776 4046 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 04:48:15 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844492, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:15.521219 4046 mini_ranger_kms.cc:384] Added ranger policy
I20260619 04:48:15.521584 4046 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAbcqnl/build/asan/bin/postgresql.jar:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAbcqnl/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 04:48:15.521647 4046 mini_ranger_kms.cc:240] Using host: 127.3.243.148
I20260619 04:48:15.540570 4046 mini_ranger_kms.cc:276] Ranger KMS PID: 16306
I20260619 04:48:15.540721 4046 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.243.148:41503
Jun 19, 2026 4:48:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 4:48:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 4:48:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 4:48:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 4:48:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 4:48:16 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 04:48:17.812928 4046 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.383s user 0.065s sys 0.318s
I20260619 04:48:17.813174 4046 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 04:48:17.813388 4046 mini_ranger_kms.cc:401] 127.3.243.148:41503/kms/v1/keys
Jun 19 04:48:17 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781844497, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 04:48:18 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844497, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.148@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
Jun 19 04:48:18 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844497, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.243.148@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
Jun 19 04:48:20 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:48:20.862300 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44613
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44613
--ranger_config_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:41503/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:21.193745 16357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:21.194077 16357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:21.194242 16357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:21.201602 16357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:48:21.201743 16357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:21.201843 16357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:21.201921 16357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:48:21.201989 16357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:48:21.212023 16357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.243.148:41503/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44613
--ranger_config_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:21.213371 16357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:21.214670 16357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:21.226413 16362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:21.226465 16363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:21.227063 16357 server_base.cc:1034] running on GCE node
W20260619 04:48:21.228387 16365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:21.229455 16357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:21.231191 16357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:21.232424 16357 hybrid_clock.cc:648] HybridClock initialized: now 1781844501232369 us; error 58 us; skew 500 ppm
I20260619 04:48:21.232841 16357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:21 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844501, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:21.236569 16357 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:48:21.238123 16357 webserver.cc:460] Webserver started at http://127.3.243.190:45963/ using document root <none> and password file <none>
I20260619 04:48:21.238843 16357 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:21.238960 16357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:21.239302 16357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:48:21 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844501, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:21.295320 16357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "f36732cf3aa844818246f1e328af8c0c"
format_stamp: "Formatted at 2026-06-19 04:48:21 on dist-test-slave-2qp8"
server_key: "56b37207902e8fa4db34a567e8267276"
server_key_iv: "51d62426f5fabf67bb20891c20a136e4"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:21.296446 16357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "f36732cf3aa844818246f1e328af8c0c"
format_stamp: "Formatted at 2026-06-19 04:48:21 on dist-test-slave-2qp8"
server_key: "56b37207902e8fa4db34a567e8267276"
server_key_iv: "51d62426f5fabf67bb20891c20a136e4"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:21.303470 16357 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260619 04:48:21.323421 16372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:21.325304 16357 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260619 04:48:21.325495 16357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "f36732cf3aa844818246f1e328af8c0c"
format_stamp: "Formatted at 2026-06-19 04:48:21 on dist-test-slave-2qp8"
server_key: "56b37207902e8fa4db34a567e8267276"
server_key_iv: "51d62426f5fabf67bb20891c20a136e4"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:21.325763 16357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:21.361438 16357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:21.366603 16357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:21.366966 16357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:21.387132 16423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44613 every 8 connection(s)
I20260619 04:48:21.387167 16357 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44613
I20260619 04:48:21.389096 16357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:48:21.393006 16424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:21.398614 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 16357
I20260619 04:48:21.398849 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:48:21.399698 4046 external_mini_cluster.cc:1428] Setting key 7c99582dba04a58ef11e8f4dc20c585c
I20260619 04:48:21.405170 16424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Bootstrap starting.
I20260619 04:48:21.410369 16424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:21.412122 16424 log.cc:824] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:21.419993 16424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: No bootstrap required, opened a new log
I20260619 04:48:21.429899 16424 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } }
I20260619 04:48:21.430296 16424 raft_consensus.cc:374] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:21.430416 16424 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36732cf3aa844818246f1e328af8c0c, State: Initialized, Role: FOLLOWER
I20260619 04:48:21.431185 16424 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } }
I20260619 04:48:21.431475 16424 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:48:21.431622 16424 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:48:21.431826 16424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 0 FOLLOWER]: Advancing to term 1
Jun 19 04:48:21 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:21.434571 16424 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } }
I20260619 04:48:21.435192 16424 leader_election.cc:302] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36732cf3aa844818246f1e328af8c0c; no voters:
I20260619 04:48:21.435949 16424 leader_election.cc:288] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:48:21.436074 16429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:48:21.445730 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:21.401930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.3.243.190:44613)
0619 04:48:21.417133 (+ 15203us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:21.417156 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:21.417213 (+ 57us) server_negotiation.cc:407] Connection header received
0619 04:48:21.419624 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:21.419650 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:21.420204 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:21.422374 (+ 2170us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:21.422936 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:21.425069 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:21.426958 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:21.428017 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:21.428221 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:21.433302 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:21.433314 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:21.433335 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:21.433394 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:21.438044 (+ 4650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:21.439251 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:21.439264 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:21.439278 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:21.439526 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:21.440094 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:21.440102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:21.440128 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:21.441067 (+ 939us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:21.442890 (+ 1823us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:21.443270 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 04:48:21.443857 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14882,"thread_start_us":14596,"threads_started":1}
I20260619 04:48:21.447975 16424 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:48:21.448406 16429 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 1 LEADER]: Becoming Leader. State: Replica: f36732cf3aa844818246f1e328af8c0c, State: Running, Role: LEADER
I20260619 04:48:21.449088 16429 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } }
I20260619 04:48:21.452962 16430 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36732cf3aa844818246f1e328af8c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } } }
I20260619 04:48:21.453307 16430 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:21.454146 16437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:48:21.453995 16431 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36732cf3aa844818246f1e328af8c0c. Latest consensus state: current_term: 1 leader_uuid: "f36732cf3aa844818246f1e328af8c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } } }
I20260619 04:48:21.454381 16431 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:21.470607 16437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:48:21.472131 16424 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 04:48:21.495199 16437 catalog_manager.cc:1347] Generated new cluster ID: 7b224538fec5481e87b5f493c4e31373
I20260619 04:48:21.495442 16437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:48:21.538373 16437 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:48:21.540364 16437 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:48:21.554536 16437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Generated new TSK 0
I20260619 04:48:21.555831 16437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:48:22.552687 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:41503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44613
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:48:22 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781844502, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 04:48:22.952956 16469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:22.953332 16469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:22.953513 16469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:22.961333 16469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:22.961468 16469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:22.961648 16469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:48:22.972332 16469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.243.148:41503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:22.973942 16469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:22.975754 16469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:22.997721 16479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:22.997753 16480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:23.005656 16482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:23.006012 16469 server_base.cc:1034] running on GCE node
I20260619 04:48:23.006933 16469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:23.009507 16469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:23.010789 16469 hybrid_clock.cc:648] HybridClock initialized: now 1781844503010721 us; error 68 us; skew 500 ppm
I20260619 04:48:23.011184 16469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:23 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:23.014531 16469 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:48:23.020534 16469 webserver.cc:460] Webserver started at http://127.3.243.129:35441/ using document root <none> and password file <none>
I20260619 04:48:23.021353 16469 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:23.021507 16469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:23.021899 16469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:48:23 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:23.044661 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "66747ca451ff4fffb10b0df7b20e71e3"
format_stamp: "Formatted at 2026-06-19 04:48:23 on dist-test-slave-2qp8"
server_key: "e062d82de674598e9e9972b8287b27d6"
server_key_iv: "f283adc5e7766c04949fe0e539ecaa35"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:23.045845 16469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "66747ca451ff4fffb10b0df7b20e71e3"
format_stamp: "Formatted at 2026-06-19 04:48:23 on dist-test-slave-2qp8"
server_key: "e062d82de674598e9e9972b8287b27d6"
server_key_iv: "f283adc5e7766c04949fe0e539ecaa35"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:23.059278 16469 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.011s sys 0.000s
I20260619 04:48:23.089370 16489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:23.091348 16469 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.009s
I20260619 04:48:23.091630 16469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "66747ca451ff4fffb10b0df7b20e71e3"
format_stamp: "Formatted at 2026-06-19 04:48:23 on dist-test-slave-2qp8"
server_key: "e062d82de674598e9e9972b8287b27d6"
server_key_iv: "f283adc5e7766c04949fe0e539ecaa35"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:23.091984 16469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:23.132838 16469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:23.138108 16469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:23.138440 16469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:23.140501 16469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:23.142527 16469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:23.142629 16469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:23.142823 16469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:23.142894 16469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 04:48:23 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844502, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:23.529593 16469 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:41793
I20260619 04:48:23.529632 16601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:41793 every 8 connection(s)
I20260619 04:48:23.531344 16469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:48:23.541432 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 16469
I20260619 04:48:23.541633 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:48:23.542397 4046 external_mini_cluster.cc:1428] Setting key ca48f207cc5e73a4b4b3589202510dfc
Jun 19 04:48:23 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844503, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:23.562817 16605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:23.535383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:48189 (local address 127.3.243.190:44613)
0619 04:48:23.543838 (+ 8455us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:23.543845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:23.543875 (+ 30us) server_negotiation.cc:407] Connection header received
0619 04:48:23.543951 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:23.543957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:23.544050 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:23.544197 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:23.546303 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:23.547369 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:23.550331 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:23.550915 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:23.551004 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:23.555505 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:23.555512 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:23.555516 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:23.555568 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:23.558988 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:23.559950 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:23.559955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:23.559958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:23.560135 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:23.560674 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:23.560679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:23.560682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:23.561062 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:23.561184 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:23.562232 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0619 04:48:23.562443 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8317,"thread_start_us":8216,"threads_started":1}
I20260619 04:48:23.565467 16604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:23.534819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44613 (local address 127.3.243.129:48189)
0619 04:48:23.540711 (+ 5892us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:23.540769 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:23.543142 (+ 2373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:23.544632 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:23.544644 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:23.545270 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:23.546105 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:23.546131 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:23.547561 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:23.547565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:23.550172 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:23.550186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:23.551320 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:23.551325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:23.551605 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:23.552441 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:23.552468 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:23.555247 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:23.559169 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:23.559178 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:23.559196 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:23.559783 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:23.560287 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:23.560291 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:23.560294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:23.560542 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:23.561296 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:23.561305 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:23.561755 (+ 450us) client_negotiation.cc:769] Sending connection context
0619 04:48:23.563402 (+ 1647us) client_negotiation.cc:240] Negotiation successful
0619 04:48:23.563993 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5568,"thread_start_us":5362,"threads_started":1}
I20260619 04:48:23.567561 16602 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44613
I20260619 04:48:23.567896 16602 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:23.568739 16602 heartbeater.cc:511] Master 127.3.243.190:44613 requested a full tablet report, sending...
I20260619 04:48:23.571194 16389 ts_manager.cc:194] Registered new tserver with Master: 66747ca451ff4fffb10b0df7b20e71e3 (127.3.243.129:41793)
I20260619 04:48:23.573276 16389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:48189
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:48:23.612043 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:41503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44613
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:48:23 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844502, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
W20260619 04:48:24.086367 16608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:24.086791 16608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:24.086994 16608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:24.098321 16608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:24.098526 16608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:24.098763 16608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:48:24.114429 16608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.243.148:41503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:24.116422 16608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:24.118160 16608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:24.133353 16623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:24.133805 16622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:24.134548 16625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:24.134706 16608 server_base.cc:1034] running on GCE node
I20260619 04:48:24.135396 16608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:24.136413 16608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:24.137638 16608 hybrid_clock.cc:648] HybridClock initialized: now 1781844504137579 us; error 53 us; skew 500 ppm
I20260619 04:48:24.137959 16608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:24 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:24.141121 16608 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:48:24.142659 16608 webserver.cc:460] Webserver started at http://127.3.243.130:34927/ using document root <none> and password file <none>
I20260619 04:48:24.143342 16608 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:24.143462 16608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:24.143770 16608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:48:24 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:24.161361 16608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "1333f5afd5b547e9ad5fa2782cbb38f3"
format_stamp: "Formatted at 2026-06-19 04:48:24 on dist-test-slave-2qp8"
server_key: "4539c10c92cb59ccdba754251215c309"
server_key_iv: "60828bd025de7d0c99384cf15dc028f6"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:24.162489 16608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "1333f5afd5b547e9ad5fa2782cbb38f3"
format_stamp: "Formatted at 2026-06-19 04:48:24 on dist-test-slave-2qp8"
server_key: "4539c10c92cb59ccdba754251215c309"
server_key_iv: "60828bd025de7d0c99384cf15dc028f6"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:24.169885 16608 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260619 04:48:24.200843 16632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:24.203001 16608 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.001s sys 0.012s
I20260619 04:48:24.203192 16608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "1333f5afd5b547e9ad5fa2782cbb38f3"
format_stamp: "Formatted at 2026-06-19 04:48:24 on dist-test-slave-2qp8"
server_key: "4539c10c92cb59ccdba754251215c309"
server_key_iv: "60828bd025de7d0c99384cf15dc028f6"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:24.203469 16608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:24.224685 16608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:24.229784 16608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:24.230136 16608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:24.231428 16608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:24.233208 16608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:24.233310 16608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:24.233454 16608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:24.233520 16608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:24.275046 16608 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:39085
I20260619 04:48:24.275070 16744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:39085 every 8 connection(s)
I20260619 04:48:24.276799 16608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:48:24.278069 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 16608
I20260619 04:48:24.278224 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:48:24.278749 4046 external_mini_cluster.cc:1428] Setting key 6f13eb26b8e173e6f18d7e0f383fe923
Jun 19 04:48:24 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:48:24.312670 16748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:24.282428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:58611 (local address 127.3.243.190:44613)
0619 04:48:24.289800 (+ 7372us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:24.289809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:24.289837 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:48:24.292453 (+ 2616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:24.292461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:24.292550 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:24.292650 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:24.294745 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.296078 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.298711 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.299450 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.299561 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:24.304850 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:24.304861 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:24.304869 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:24.304919 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:24.308991 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:24.310125 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:24.310133 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:24.310138 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:24.310289 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:24.310750 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:24.310757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:24.310763 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:24.311126 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:24.311297 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:24.311976 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 04:48:24.312260 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7210,"thread_start_us":7081,"threads_started":1}
I20260619 04:48:24.313671 16747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:24.281833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44613 (local address 127.3.243.130:58611)
0619 04:48:24.288696 (+ 6863us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:24.288747 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:24.291943 (+ 3196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:24.293083 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:24.293098 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:24.293734 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:24.294546 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:24.294574 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.296321 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.296329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:24.298518 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:24.298536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.299934 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.299943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:24.300335 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:24.301156 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:24.301190 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:24.304543 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:24.309169 (+ 4626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:24.309178 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:24.309305 (+ 127us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:24.309931 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:24.310441 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:24.310444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:24.310446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:24.310629 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:24.311249 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:24.311257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:24.311595 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 04:48:24.311869 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 04:48:24.312419 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6551,"thread_start_us":6325,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
I20260619 04:48:24.316736 16745 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44613
I20260619 04:48:24.317046 16745 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:24.318122 16745 heartbeater.cc:511] Master 127.3.243.190:44613 requested a full tablet report, sending...
I20260619 04:48:24.319949 16389 ts_manager.cc:194] Registered new tserver with Master: 1333f5afd5b547e9ad5fa2782cbb38f3 (127.3.243.130:39085)
I20260619 04:48:24.321218 16389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:58611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:48:24.345763 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:41503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44613
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:48:24.577445 16602 heartbeater.cc:503] Master 127.3.243.190:44613 was elected leader, sending a full tablet report...
W20260619 04:48:24.689322 16751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:24.689569 16751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:24.689654 16751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:24.697484 16751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:24.697580 16751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:24.697687 16751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:48:24.707192 16751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.243.148:41503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:24.708577 16751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:24.709878 16751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:24.722306 16756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:24.722335 16759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:24.722409 16757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:24.723379 16751 server_base.cc:1034] running on GCE node
I20260619 04:48:24.723891 16751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:24.724764 16751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:24.725982 16751 hybrid_clock.cc:648] HybridClock initialized: now 1781844504725948 us; error 51 us; skew 500 ppm
I20260619 04:48:24.726361 16751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:24 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:24.729524 16751 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:48:24.731040 16751 webserver.cc:460] Webserver started at http://127.3.243.131:43869/ using document root <none> and password file <none>
I20260619 04:48:24.731766 16751 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:24.731875 16751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:24.732271 16751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 04:48:24 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:24.746860 16751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "749f8975157d4c6280cdc30ee93a801d"
format_stamp: "Formatted at 2026-06-19 04:48:24 on dist-test-slave-2qp8"
server_key: "107a2c49177c0d46182c7a4e18442310"
server_key_iv: "2d2484e06a598a60be61fe172352132d"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:24.747620 16751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "749f8975157d4c6280cdc30ee93a801d"
format_stamp: "Formatted at 2026-06-19 04:48:24 on dist-test-slave-2qp8"
server_key: "107a2c49177c0d46182c7a4e18442310"
server_key_iv: "2d2484e06a598a60be61fe172352132d"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:24.753152 16751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:48:24.769762 16766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:24.771289 16751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:48:24.771451 16751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "749f8975157d4c6280cdc30ee93a801d"
format_stamp: "Formatted at 2026-06-19 04:48:24 on dist-test-slave-2qp8"
server_key: "107a2c49177c0d46182c7a4e18442310"
server_key_iv: "2d2484e06a598a60be61fe172352132d"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:24.771662 16751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:24.789261 16751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:24.793429 16751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:24.793709 16751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:24.794646 16751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:24.795915 16751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:24.795984 16751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:24.796183 16751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:24.796262 16751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 04:48:24.828058 16751 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:41911
I20260619 04:48:24.828092 16878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:41911 every 8 connection(s)
I20260619 04:48:24.829493 16751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:48:24.839545 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 16751
I20260619 04:48:24.839728 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:48:24.840266 4046 external_mini_cluster.cc:1428] Setting key 3a5006633d56276c32065064326e093a
Jun 19 04:48:24 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844504, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:24.848389 16881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:24.831852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:50267 (local address 127.3.243.190:44613)
0619 04:48:24.832336 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:24.832342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:24.833311 (+ 969us) server_negotiation.cc:407] Connection header received
0619 04:48:24.834947 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:24.834951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:24.835027 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:24.835121 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:24.836500 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.837289 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.838722 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.839144 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.839282 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:24.843070 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:24.843076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:24.843080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:24.843124 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:24.845818 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:24.846466 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:24.846470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:24.846472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:24.846584 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:24.846955 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:24.846958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:24.846960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:24.847215 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:24.847327 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:24.847876 (+ 549us) server_negotiation.cc:299] Negotiation successful
0619 04:48:24.848025 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
I20260619 04:48:24.849359 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:24.832507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44613 (local address 127.3.243.131:50267)
0619 04:48:24.833154 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:24.833195 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:24.834673 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:24.835318 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:24.835325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:24.835706 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:24.836360 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:24.836379 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.837420 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.837424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:24.838609 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:24.838620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.839349 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.839354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:24.839640 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:24.840425 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:24.840458 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:24.842868 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:24.845928 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:24.845935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:24.845946 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:24.846332 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:24.846669 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:24.846672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:24.846675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:24.846862 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:24.847320 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:24.847325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:24.847589 (+ 264us) client_negotiation.cc:769] Sending connection context
0619 04:48:24.847793 (+ 204us) client_negotiation.cc:240] Negotiation successful
0619 04:48:24.848213 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":264,"threads_started":1}
I20260619 04:48:24.850690 16879 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44613
I20260619 04:48:24.850978 16879 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:24.851651 16879 heartbeater.cc:511] Master 127.3.243.190:44613 requested a full tablet report, sending...
I20260619 04:48:24.852994 16389 ts_manager.cc:194] Registered new tserver with Master: 749f8975157d4c6280cdc30ee93a801d (127.3.243.131:41911)
I20260619 04:48:24.853940 16389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:50267
I20260619 04:48:24.855794 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:48:24.857577 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 16469
Jun 19 04:48:24 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:48:24.873430 16884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:24.858546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47908 (local address 127.3.243.129:41793)
0619 04:48:24.859145 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:24.859151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:24.859173 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:48:24.859231 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:24.859237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:24.859415 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:24.859533 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:24.861094 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.862062 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.863562 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.864080 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.864241 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:24.868191 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:24.868201 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:24.868214 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:24.868247 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:24.871041 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:24.871734 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:24.871740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:24.871742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:24.871889 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:24.872258 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:24.872262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:24.872264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:24.872555 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:24.872778 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:24.872985 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 04:48:24.873166 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":325,"threads_started":1}
I20260619 04:48:24.874697 16597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:24.957962 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 16469
I20260619 04:48:24.973654 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 16608
Jun 19 04:48:24 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:48:24.989048 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:24.974574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.3.243.130:39085)
0619 04:48:24.975197 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:24.975204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:24.975229 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:48:24.975294 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:24.975300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:24.975498 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:24.975704 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:24.977095 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.978109 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.979329 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:24.979860 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:24.979951 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:24.983881 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:24.983890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:24.983903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:24.983947 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:24.986575 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:24.987197 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:24.987203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:24.987205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:24.987354 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:24.987650 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:24.987653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:24.987655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:24.987973 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:24.988340 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:24.988557 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 04:48:24.988743 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":356,"threads_started":1}
I20260619 04:48:24.990267 16740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:25.071208 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 16608
I20260619 04:48:25.086571 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 16751
Jun 19 04:48:25 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844500, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:48:25.103483 16888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:25.087570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.3.243.131:41911)
0619 04:48:25.088239 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:25.088246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:25.088274 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:48:25.088349 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:25.088355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:25.088602 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:25.088769 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:25.090137 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:25.091150 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:25.092609 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:25.093177 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:25.093316 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:25.097860 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:25.097870 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:25.097885 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:25.097920 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:25.100698 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:25.101332 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:25.101342 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:25.101352 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:25.101472 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:25.101855 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:25.101861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:25.101864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:25.102243 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:25.102557 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:25.102859 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 04:48:25.103113 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":328,"threads_started":1}
I20260619 04:48:25.104856 16874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:25.187248 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 16751
I20260619 04:48:25.202863 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 16357
I20260619 04:48:25.204301 16419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:25.309716 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 16357
I20260619 04:48:25.331487 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44613
--webserver_interface=127.3.243.190
--webserver_port=45963
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44613
--ranger_config_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:41503/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.243.148:41503/kms with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:25.659091 16891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:25.659364 16891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:25.659451 16891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:25.666560 16891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:48:25.666658 16891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:25.666716 16891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:25.666783 16891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:48:25.666831 16891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:48:25.676710 16891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.243.148:41503/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44613
--ranger_config_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=45963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:25.677924 16891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:25.679168 16891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:25.690984 16899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:25.691026 16897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:25.691051 16896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:25.691736 16891 server_base.cc:1034] running on GCE node
I20260619 04:48:25.692481 16891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:25.693755 16891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:25.694962 16891 hybrid_clock.cc:648] HybridClock initialized: now 1781844505694932 us; error 29 us; skew 500 ppm
I20260619 04:48:25.695266 16891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:25 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:25.698295 16891 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:48:25.699774 16891 webserver.cc:460] Webserver started at http://127.3.243.190:45963/ using document root <none> and password file <none>
I20260619 04:48:25.700572 16891 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:25.700707 16891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 04:48:25 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844505, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:25.750774 16891 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 04:48:25.754056 16907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:25.755688 16891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:48:25.755882 16891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "f36732cf3aa844818246f1e328af8c0c"
format_stamp: "Formatted at 2026-06-19 04:48:21 on dist-test-slave-2qp8"
server_key: "56b37207902e8fa4db34a567e8267276"
server_key_iv: "51d62426f5fabf67bb20891c20a136e4"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:25.756897 16891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:25.791776 16891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:25.795802 16891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:25.796051 16891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:25.814575 16891 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44613
I20260619 04:48:25.814539 16958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44613 every 8 connection(s)
I20260619 04:48:25.816092 16891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:48:25.822743 16959 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:48:25.825881 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 16891
I20260619 04:48:25.825982 16959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Bootstrap starting.
I20260619 04:48:25.826740 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:41503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:41793
--local_ip_for_outbound_sockets=127.3.243.129
--tserver_master_addrs=127.3.243.190:44613
--webserver_port=35441
--webserver_interface=127.3.243.129
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.243.148:41503/kms with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:48:25.844925 16959 log.cc:824] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:25.854118 16959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:48:25.854628 16959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Bootstrap complete.
I20260619 04:48:25.863598 16959 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } }
I20260619 04:48:25.864619 16959 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36732cf3aa844818246f1e328af8c0c, State: Initialized, Role: FOLLOWER
I20260619 04:48:25.865208 16959 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } }
I20260619 04:48:25.865458 16959 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:48:25.865593 16959 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:48:25.865772 16959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:48:25.869516 16959 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } }
I20260619 04:48:25.869997 16959 leader_election.cc:302] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36732cf3aa844818246f1e328af8c0c; no voters:
I20260619 04:48:25.870663 16959 leader_election.cc:288] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 04:48:25.870805 16963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:48:25.871991 16963 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [term 2 LEADER]: Becoming Leader. State: Replica: f36732cf3aa844818246f1e328af8c0c, State: Running, Role: LEADER
I20260619 04:48:25.872624 16963 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } }
I20260619 04:48:25.872855 16959 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:48:25.875356 16965 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36732cf3aa844818246f1e328af8c0c. Latest consensus state: current_term: 2 leader_uuid: "f36732cf3aa844818246f1e328af8c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } } }
I20260619 04:48:25.875782 16965 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:25.876145 16964 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f36732cf3aa844818246f1e328af8c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36732cf3aa844818246f1e328af8c0c" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44613 } } }
I20260619 04:48:25.876528 16964 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:25.888345 16972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:48:25.893869 16972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:48:25.895229 16972 catalog_manager.cc:1259] Loaded cluster ID: 7b224538fec5481e87b5f493c4e31373
I20260619 04:48:25.895388 16972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:48:25.899111 16972 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:48:25.900827 16972 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Loaded TSK: 0
I20260619 04:48:25.902459 16972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:48:25.909209 16959 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 04:48:26.253517 16961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:26.253913 16961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:26.254081 16961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:26.263677 16961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:26.263829 16961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:26.264037 16961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:48:26.279875 16961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.243.148:41503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:41793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=35441
--tserver_master_addrs=127.3.243.190:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:26.281657 16961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:26.283452 16961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:26.322295 16996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:26.326143 16997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:26.327960 16961 server_base.cc:1034] running on GCE node
W20260619 04:48:26.328354 16999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:26.329115 16961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:26.330302 16961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:26.331527 16961 hybrid_clock.cc:648] HybridClock initialized: now 1781844506331460 us; error 55 us; skew 500 ppm
I20260619 04:48:26.331858 16961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:26 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844506, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:26.336035 16961 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:48:26.351121 16961 webserver.cc:460] Webserver started at http://127.3.243.129:35441/ using document root <none> and password file <none>
I20260619 04:48:26.352027 16961 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:26.352281 16961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 04:48:26 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844506, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:26.402196 16961 fs_manager.cc:709] Time spent opening directory manager: real 0.022s user 0.011s sys 0.008s
I20260619 04:48:26.413127 17007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:26.415167 16961 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.008s
I20260619 04:48:26.415366 16961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "66747ca451ff4fffb10b0df7b20e71e3"
format_stamp: "Formatted at 2026-06-19 04:48:23 on dist-test-slave-2qp8"
server_key: "e062d82de674598e9e9972b8287b27d6"
server_key_iv: "f283adc5e7766c04949fe0e539ecaa35"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:26.416838 16961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:26.492612 16961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:26.496874 16961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:26.497162 16961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:26.498245 16961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:26.500401 16961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:26.500522 16961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:26.500705 16961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:26.500806 16961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 04:48:26.572147 16961 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:41793
I20260619 04:48:26.572207 17120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:41793 every 8 connection(s)
I20260619 04:48:26.573758 16961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:48:26.578537 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 16961
I20260619 04:48:26.579345 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:41503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:39085
--local_ip_for_outbound_sockets=127.3.243.130
--tserver_master_addrs=127.3.243.190:44613
--webserver_port=34927
--webserver_interface=127.3.243.130
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.243.148:41503/kms with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:48:26 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844506, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:26.605918 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:26.577358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:50287 (local address 127.3.243.190:44613)
0619 04:48:26.579417 (+ 2059us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:26.579759 (+ 342us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:26.583084 (+ 3325us) server_negotiation.cc:407] Connection header received
0619 04:48:26.583525 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:26.583546 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:26.584058 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:26.584519 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:26.586954 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:26.588938 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:26.591631 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:26.592729 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:26.592902 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:26.597003 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:26.597010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:26.597025 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:26.597061 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:26.600677 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:26.601604 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:26.601619 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:26.601624 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:26.601776 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:26.602363 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:26.602369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:26.602372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:26.603038 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:26.603274 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:26.604071 (+ 797us) server_negotiation.cc:299] Negotiation successful
0619 04:48:26.604478 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1791,"thread_start_us":325,"threads_started":1}
I20260619 04:48:26.607999 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:26.577423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44613 (local address 127.3.243.129:50287)
0619 04:48:26.580028 (+ 2605us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:26.580075 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:26.582559 (+ 2484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:26.585238 (+ 2679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:26.585251 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:26.585925 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:26.586715 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:26.586741 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:26.589179 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:26.589185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:26.591473 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:26.591489 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:26.592957 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:26.592965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:26.593251 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:26.594070 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:26.594094 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:26.596737 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:26.600833 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:26.600842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:26.600858 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:26.601456 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:26.601899 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:26.601905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:26.601908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:26.602252 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:26.603193 (+ 941us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:26.603203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:26.603545 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 04:48:26.603838 (+ 293us) client_negotiation.cc:240] Negotiation successful
0619 04:48:26.604446 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":702,"threads_started":1}
I20260619 04:48:26.611686 17121 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44613
I20260619 04:48:26.612021 17121 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:26.612964 17121 heartbeater.cc:511] Master 127.3.243.190:44613 requested a full tablet report, sending...
W20260619 04:48:26.616232 17121 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:44613 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 04:48:26.616459 17121 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:26.616739 17121 heartbeater.cc:511] Master 127.3.243.190:44613 requested a full tablet report, sending...
I20260619 04:48:26.617874 17121 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:26.618149 17121 heartbeater.cc:511] Master 127.3.243.190:44613 requested a full tablet report, sending...
W20260619 04:48:26.619203 17121 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:48:26.994606 17125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:26.994880 17125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:26.995038 17125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:27.002384 17125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:27.002490 17125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:27.002656 17125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:48:27.014147 17125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.243.148:41503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:39085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=34927
--tserver_master_addrs=127.3.243.190:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:27.015849 17125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:27.017382 17125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:27.035322 17139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:27.035977 17140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:27.036284 17142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:27.037081 17125 server_base.cc:1034] running on GCE node
I20260619 04:48:27.037765 17125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:27.042130 17125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:27.043411 17125 hybrid_clock.cc:648] HybridClock initialized: now 1781844507043365 us; error 40 us; skew 500 ppm
I20260619 04:48:27.043749 17125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:27 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:27.046978 17125 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:48:27.049049 17125 webserver.cc:460] Webserver started at http://127.3.243.130:34927/ using document root <none> and password file <none>
I20260619 04:48:27.049836 17125 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:27.049996 17125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 04:48:27 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:27.071659 17125 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:48:27.092723 17150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:27.094767 17125 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.000s sys 0.017s
I20260619 04:48:27.095019 17125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "1333f5afd5b547e9ad5fa2782cbb38f3"
format_stamp: "Formatted at 2026-06-19 04:48:24 on dist-test-slave-2qp8"
server_key: "4539c10c92cb59ccdba754251215c309"
server_key_iv: "60828bd025de7d0c99384cf15dc028f6"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:27.096582 17125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:27.149011 17125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:27.154661 17125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:27.155042 17125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:27.156418 17125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:27.158250 17125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:27.158419 17125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 04:48:27.158572 17125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:27.158687 17125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:27.197660 17125 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:39085
I20260619 04:48:27.197728 17264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:39085 every 8 connection(s)
I20260619 04:48:27.199960 17125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:48:27.205636 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 17125
I20260619 04:48:27.206672 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.243.148:41503/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:41911
--local_ip_for_outbound_sockets=127.3.243.131
--tserver_master_addrs=127.3.243.190:44613
--webserver_port=43869
--webserver_interface=127.3.243.131
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.243.148:41503/kms with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:48:27 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781844507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 04:48:27 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:27.243623 17268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:27.203791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44613 (local address 127.3.243.130:53027)
0619 04:48:27.215613 (+ 11822us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:27.215673 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:27.218799 (+ 3126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:27.222923 (+ 4124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:27.222936 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:27.223638 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:27.224537 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:27.224562 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.227061 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.227067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:27.229376 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:27.229459 (+ 83us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.230664 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.230672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:27.230975 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:27.231810 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:27.231841 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:27.234549 (+ 2708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:27.238797 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:27.238806 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:27.238825 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:27.239424 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:27.239864 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:27.239871 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:27.239874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:27.240166 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:27.240856 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:27.240865 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:27.241335 (+ 470us) client_negotiation.cc:769] Sending connection context
0619 04:48:27.241657 (+ 322us) client_negotiation.cc:240] Negotiation successful
0619 04:48:27.242244 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11481,"thread_start_us":11238,"threads_started":1}
I20260619 04:48:27.245289 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:27.203210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:53027 (local address 127.3.243.190:44613)
0619 04:48:27.222273 (+ 19063us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:27.222279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:27.222302 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:48:27.222371 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:27.222376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:27.222460 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:27.222609 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:27.224755 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.226832 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.229650 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.230382 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.233260 (+ 2878us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:27.234806 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:27.234812 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:27.234817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:27.234857 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:27.238614 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:27.239584 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:27.239590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:27.239593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:27.239731 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:27.240334 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:27.240340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:27.240344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:27.240698 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:27.244512 (+ 3814us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:27.244645 (+ 133us) server_negotiation.cc:299] Negotiation successful
0619 04:48:27.244884 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18911,"thread_start_us":19519,"threads_started":1}
I20260619 04:48:27.246646 17265 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44613
I20260619 04:48:27.246953 17265 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:27.247768 17265 heartbeater.cc:511] Master 127.3.243.190:44613 requested a full tablet report, sending...
I20260619 04:48:27.249881 16924 ts_manager.cc:194] Registered new tserver with Master: 1333f5afd5b547e9ad5fa2782cbb38f3 (127.3.243.130:39085)
I20260619 04:48:27.252043 16924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:53027
I20260619 04:48:27.619359 17121 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:27.619782 17121 heartbeater.cc:511] Master 127.3.243.190:44613 requested a full tablet report, sending...
I20260619 04:48:27.621069 16924 ts_manager.cc:194] Registered new tserver with Master: 66747ca451ff4fffb10b0df7b20e71e3 (127.3.243.129:41793)
I20260619 04:48:27.622342 16924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:50287
W20260619 04:48:27.654131 17269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:27.654486 17269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:27.654704 17269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:27.662508 17269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:27.662642 17269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:27.662842 17269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:48:27.674926 17269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.243.148:41503/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:41911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=43869
--tserver_master_addrs=127.3.243.190:44613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:27.676313 17269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:27.677652 17269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:27.701768 17275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Jun 19 04:48:27 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:27.706673 17269 server_base.cc:1034] running on GCE node
W20260619 04:48:27.706979 17276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:27.707331 17278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:27.708051 17269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:27.709220 17269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:27.710448 17269 hybrid_clock.cc:648] HybridClock initialized: now 1781844507710370 us; error 59 us; skew 500 ppm
I20260619 04:48:27.710860 17269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:27 dist-test-slave-2qp8 krb5kdc[15913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:27.715377 17269 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:48:27.733836 17269 webserver.cc:460] Webserver started at http://127.3.243.131:43869/ using document root <none> and password file <none>
I20260619 04:48:27.734655 17269 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:27.734822 17269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 04:48:27 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:27.783516 17269 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.011s sys 0.005s
I20260619 04:48:27.796753 17286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:27.798921 17269 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.010s
I20260619 04:48:27.799155 17269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "749f8975157d4c6280cdc30ee93a801d"
format_stamp: "Formatted at 2026-06-19 04:48:24 on dist-test-slave-2qp8"
server_key: "107a2c49177c0d46182c7a4e18442310"
server_key_iv: "2d2484e06a598a60be61fe172352132d"
server_key_version: "kuduclusterkey@0"
I20260619 04:48:27.800614 17269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:27.831103 17269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:27.835393 17269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:27.835686 17269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:27.836845 17269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:27.839146 17269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:27.839268 17269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:27.839409 17269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:27.839495 17269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:27.888470 17269 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:41911
I20260619 04:48:27.888532 17398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:41911 every 8 connection(s)
I20260619 04:48:27.890364 17269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:48:27.893872 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 17269
I20260619 04:48:27.923972 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:27.895334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43718 (local address 127.3.243.190:44613)
0619 04:48:27.900575 (+ 5241us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:27.900584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:27.905633 (+ 5049us) server_negotiation.cc:407] Connection header received
0619 04:48:27.905784 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:27.905793 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:27.905936 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:27.906091 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:27.909542 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.910694 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.912644 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.913291 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.913404 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:27.916840 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:27.916847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:27.916851 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:27.916887 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:27.920243 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:27.921093 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:27.921099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:27.921102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:27.921239 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:27.921664 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:27.921668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:27.921671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:27.922024 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:27.922259 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:27.923390 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0619 04:48:27.923589 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5059,"thread_start_us":4916,"threads_started":1}
Jun 19 04:48:27 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:27.944715 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:27.902891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44613 (local address 127.3.243.131:34771)
0619 04:48:27.908509 (+ 5618us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:27.908565 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:27.910905 (+ 2340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:27.925988 (+ 15083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:27.926003 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:27.926649 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:27.927830 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:27.927859 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.929486 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.929492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:27.931521 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:27.931535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.932718 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.932726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:27.933009 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:27.933749 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:27.933778 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:27.936404 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:27.940288 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:27.940296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:27.940320 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:27.940860 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:27.941253 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:27.941257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:27.941260 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:27.941506 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:27.942119 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:27.942126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:27.942533 (+ 407us) client_negotiation.cc:769] Sending connection context
0619 04:48:27.942830 (+ 297us) client_negotiation.cc:240] Negotiation successful
0619 04:48:27.943347 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5261,"thread_start_us":5037,"threads_started":1}
I20260619 04:48:27.945719 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:27.903780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:34771 (local address 127.3.243.190:44613)
0619 04:48:27.924570 (+ 20790us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:27.924575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:27.924594 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:48:27.924645 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:27.924650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:27.924713 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:27.924811 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:27.928069 (+ 3258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.929299 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.931684 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.932365 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.932477 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:27.936643 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:27.936649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:27.936653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:27.936690 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:27.940114 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:27.940987 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:27.940992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:27.940996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:27.941132 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:27.941619 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:27.941624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:27.941627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:27.941971 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:27.942120 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:27.945186 (+ 3066us) server_negotiation.cc:299] Negotiation successful
0619 04:48:27.945392 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20631,"thread_start_us":20556,"threads_started":1}
I20260619 04:48:27.947427 17399 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44613
I20260619 04:48:27.947781 17399 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:27.948566 17399 heartbeater.cc:511] Master 127.3.243.190:44613 requested a full tablet report, sending...
I20260619 04:48:27.950168 16924 ts_manager.cc:194] Registered new tserver with Master: 749f8975157d4c6280cdc30ee93a801d (127.3.243.131:41911)
I20260619 04:48:27.951229 16924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:34771
I20260619 04:48:27.951721 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:48:27.975772 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:27.958138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.3.243.190:44613)
0619 04:48:27.958295 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:27.958300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:27.958854 (+ 554us) server_negotiation.cc:407] Connection header received
0619 04:48:27.959042 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:27.959058 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:27.959137 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:27.959227 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:27.961436 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.962382 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.964820 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:27.965393 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:27.965512 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:27.968808 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:27.968815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:27.968818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:27.968852 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:27.972248 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:27.973101 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:27.973106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:27.973110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:27.973248 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:27.973777 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:27.973782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:27.973785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:27.974175 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:27.974407 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:27.975067 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 04:48:27.975371 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:48:27.984885 16924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:48:27.997614 16924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 19 04:48:28 dist-test-slave-2qp8 krb5kdc[15913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844507, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for HTTP/127.3.243.148@KRBTEST.COM
I20260619 04:48:28.046994 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.017696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:41911 (local address 127.0.0.1:48540)
0619 04:48:28.033510 (+ 15814us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.033533 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:28.033688 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:28.036797 (+ 3109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:28.036801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:28.036837 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:28.037228 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.037243 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.040525 (+ 3282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.040530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.043826 (+ 3296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.043846 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.046001 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.046007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.046202 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.046276 (+ 74us) client_negotiation.cc:769] Sending connection context
0619 04:48:28.046420 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 04:48:28.046604 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15661,"mutex_wait_us":59,"thread_start_us":13239,"threads_started":1}
I20260619 04:48:28.047820 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.018309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.3.243.131:41911)
0619 04:48:28.032581 (+ 14272us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:28.032594 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:28.036223 (+ 3629us) server_negotiation.cc:407] Connection header received
0619 04:48:28.036343 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:28.036350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:28.036610 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:28.036787 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:28.037480 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.040354 (+ 2874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.044024 (+ 3670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.045508 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.045630 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.045858 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:28.047343 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0619 04:48:28.047504 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14041,"thread_start_us":16680,"threads_started":1}
I20260619 04:48:28.051741 17334 tablet_service.cc:1467] Processing CreateTablet for tablet 070a3ff40f5f4c0393c0441c3cf77a47 (DEFAULT_TABLE table=test-table [id=3f2786a0802c4078a9375f47884223d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:48:28.053056 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.033154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.3.243.129:41793)
0619 04:48:28.034274 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:28.034284 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:28.035094 (+ 810us) server_negotiation.cc:407] Connection header received
0619 04:48:28.035245 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:28.035257 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:28.035601 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:28.035833 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:28.037524 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.039269 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.049602 (+ 10333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.051096 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.051226 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.051483 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:28.052544 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0619 04:48:28.052701 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":747,"threads_started":1}
I20260619 04:48:28.053185 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070a3ff40f5f4c0393c0441c3cf77a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:28.053436 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.033480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.0.0.1:55986)
0619 04:48:28.034783 (+ 1303us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.034822 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:28.034959 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:28.036090 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:28.036095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:28.036181 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:28.036624 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.036640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.039446 (+ 2806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.039450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.041773 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.041788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.051628 (+ 9840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.051634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.051797 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.051828 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:48:28.051917 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 04:48:28.052059 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":2,"thread_start_us":978,"threads_started":1}
I20260619 04:48:28.057245 17056 tablet_service.cc:1467] Processing CreateTablet for tablet 070a3ff40f5f4c0393c0441c3cf77a47 (DEFAULT_TABLE table=test-table [id=3f2786a0802c4078a9375f47884223d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:48:28.058553 17056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070a3ff40f5f4c0393c0441c3cf77a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:28.079069 17422 tablet_bootstrap.cc:492] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3: Bootstrap starting.
I20260619 04:48:28.084192 17422 tablet_bootstrap.cc:654] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:28.085421 17422 log.cc:824] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:28.088660 17422 tablet_bootstrap.cc:492] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3: No bootstrap required, opened a new log
I20260619 04:48:28.089039 17422 ts_tablet_manager.cc:1397] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 04:48:28.094708 17418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.016224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:39085 (local address 127.0.0.1:51196)
0619 04:48:28.031489 (+ 15265us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.031553 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:28.031786 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:28.041939 (+ 10153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:28.041944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:28.041968 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:28.042391 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.042405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.074067 (+ 31662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.074074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.076252 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.076271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.093856 (+ 17585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.093864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.094071 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.094107 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:48:28.094194 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 04:48:28.094365 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14881,"thread_start_us":18927,"threads_started":1}
I20260619 04:48:28.095646 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.016333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.3.243.130:39085)
0619 04:48:28.038899 (+ 22566us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:28.038907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:28.038935 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:48:28.039028 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:28.039034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:28.039242 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:28.039432 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:28.042638 (+ 3206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.049401 (+ 6763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.076493 (+ 27092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.093235 (+ 16742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.093427 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.093642 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:28.095190 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0619 04:48:28.095373 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22396,"thread_start_us":23423,"threads_started":1}
I20260619 04:48:28.099543 17200 tablet_service.cc:1467] Processing CreateTablet for tablet 070a3ff40f5f4c0393c0441c3cf77a47 (DEFAULT_TABLE table=test-table [id=3f2786a0802c4078a9375f47884223d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:48:28.100984 17200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070a3ff40f5f4c0393c0441c3cf77a47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:28.102064 17422 raft_consensus.cc:348] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } }
I20260619 04:48:28.102684 17422 raft_consensus.cc:374] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:28.102844 17422 raft_consensus.cc:729] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66747ca451ff4fffb10b0df7b20e71e3, State: Initialized, Role: FOLLOWER
I20260619 04:48:28.104038 17422 consensus_queue.cc:260] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } }
I20260619 04:48:28.125865 17121 heartbeater.cc:503] Master 127.3.243.190:44613 was elected leader, sending a full tablet report...
I20260619 04:48:28.126682 17422 ts_tablet_manager.cc:1428] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3: Time spent starting tablet: real 0.037s user 0.014s sys 0.016s
I20260619 04:48:28.140908 17425 tablet_bootstrap.cc:492] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: Bootstrap starting.
I20260619 04:48:28.150372 17425 tablet_bootstrap.cc:654] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:28.151746 17425 log.cc:824] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:28.158772 17426 tablet_bootstrap.cc:492] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: Bootstrap starting.
I20260619 04:48:28.163069 17426 tablet_bootstrap.cc:654] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:28.164435 17426 log.cc:824] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:28.167737 17425 tablet_bootstrap.cc:492] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: No bootstrap required, opened a new log
I20260619 04:48:28.168023 17425 ts_tablet_manager.cc:1397] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20260619 04:48:28.179093 17426 tablet_bootstrap.cc:492] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: No bootstrap required, opened a new log
I20260619 04:48:28.179404 17426 ts_tablet_manager.cc:1397] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260619 04:48:28.200568 17425 raft_consensus.cc:348] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } }
I20260619 04:48:28.201116 17425 raft_consensus.cc:374] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:28.201217 17425 raft_consensus.cc:729] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749f8975157d4c6280cdc30ee93a801d, State: Initialized, Role: FOLLOWER
I20260619 04:48:28.216722 17425 consensus_queue.cc:260] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } }
I20260619 04:48:28.221832 17426 raft_consensus.cc:348] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } }
I20260619 04:48:28.223523 17426 raft_consensus.cc:374] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:28.223629 17426 raft_consensus.cc:729] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1333f5afd5b547e9ad5fa2782cbb38f3, State: Initialized, Role: FOLLOWER
I20260619 04:48:28.224339 17426 consensus_queue.cc:260] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } }
I20260619 04:48:28.225489 17399 heartbeater.cc:503] Master 127.3.243.190:44613 was elected leader, sending a full tablet report...
I20260619 04:48:28.229957 17425 ts_tablet_manager.cc:1428] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: Time spent starting tablet: real 0.062s user 0.014s sys 0.003s
I20260619 04:48:28.236876 17265 heartbeater.cc:503] Master 127.3.243.190:44613 was elected leader, sending a full tablet report...
I20260619 04:48:28.237694 17426 ts_tablet_manager.cc:1428] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: Time spent starting tablet: real 0.058s user 0.016s sys 0.010s
I20260619 04:48:28.242048 17430 raft_consensus.cc:482] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:28.242295 17430 raft_consensus.cc:504] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } }
I20260619 04:48:28.250680 17430 leader_election.cc:288] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 749f8975157d4c6280cdc30ee93a801d (127.3.243.131:41911), 66747ca451ff4fffb10b0df7b20e71e3 (127.3.243.129:41793)
I20260619 04:48:28.262941 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.246626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:42493 (local address 127.3.243.129:41793)
0619 04:48:28.246802 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:28.246807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:28.256048 (+ 9241us) server_negotiation.cc:407] Connection header received
0619 04:48:28.256359 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:28.256365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:28.256447 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:28.256508 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:28.257241 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.258282 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.260604 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.261707 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.261856 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.261929 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:28.262534 (+ 605us) server_negotiation.cc:299] Negotiation successful
0619 04:48:28.262666 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:48:28.263880 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.246101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.3.243.130:42493)
0619 04:48:28.255939 (+ 9838us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.255959 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:28.256170 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:28.256684 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:28.256688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:28.256712 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:28.257060 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.257072 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.258460 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.258465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.260443 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.260456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.262162 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.262168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.262342 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.262378 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:48:28.263411 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0619 04:48:28.263547 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9712,"thread_start_us":9532,"threads_started":1}
I20260619 04:48:28.265144 17076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070a3ff40f5f4c0393c0441c3cf77a47" candidate_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66747ca451ff4fffb10b0df7b20e71e3" is_pre_election: true
I20260619 04:48:28.265681 17076 raft_consensus.cc:2454] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1333f5afd5b547e9ad5fa2782cbb38f3 in term 0.
I20260619 04:48:28.266388 17152 leader_election.cc:302] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1333f5afd5b547e9ad5fa2782cbb38f3, 66747ca451ff4fffb10b0df7b20e71e3; no voters:
I20260619 04:48:28.267079 17430 raft_consensus.cc:2781] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:48:28.267232 17430 raft_consensus.cc:482] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:48:28.267330 17430 raft_consensus.cc:3037] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:28.269346 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.244423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:34527 (local address 127.3.243.131:41911)
0619 04:48:28.244630 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:28.244637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:28.250824 (+ 6187us) server_negotiation.cc:407] Connection header received
0619 04:48:28.251206 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:28.251212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:28.251291 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:28.251358 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:28.252263 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.253404 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.261795 (+ 8391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.263000 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.263127 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.263191 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:28.268983 (+ 5792us) server_negotiation.cc:299] Negotiation successful
0619 04:48:28.269124 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:48:28.270190 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.244377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:41911 (local address 127.3.243.130:34527)
0619 04:48:28.250695 (+ 6318us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.250716 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:28.251018 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:28.251572 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:28.251576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:28.251603 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:28.252072 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.252087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.253634 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.253643 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.261618 (+ 7975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:28.261632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.268581 (+ 6949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.268587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:28.268789 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.268825 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:48:28.269697 (+ 872us) client_negotiation.cc:240] Negotiation successful
0619 04:48:28.269853 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6164,"thread_start_us":6002,"threads_started":1}
I20260619 04:48:28.270402 17430 raft_consensus.cc:504] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } }
I20260619 04:48:28.271168 17430 leader_election.cc:288] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [CANDIDATE]: Term 1 election: Requested vote from peers 749f8975157d4c6280cdc30ee93a801d (127.3.243.131:41911), 66747ca451ff4fffb10b0df7b20e71e3 (127.3.243.129:41793)
I20260619 04:48:28.271294 17354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070a3ff40f5f4c0393c0441c3cf77a47" candidate_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f8975157d4c6280cdc30ee93a801d" is_pre_election: true
I20260619 04:48:28.271759 17076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070a3ff40f5f4c0393c0441c3cf77a47" candidate_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66747ca451ff4fffb10b0df7b20e71e3"
I20260619 04:48:28.271873 17354 raft_consensus.cc:2454] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1333f5afd5b547e9ad5fa2782cbb38f3 in term 0.
I20260619 04:48:28.271960 17076 raft_consensus.cc:3037] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:28.272419 17353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070a3ff40f5f4c0393c0441c3cf77a47" candidate_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f8975157d4c6280cdc30ee93a801d"
I20260619 04:48:28.272678 17353 raft_consensus.cc:3037] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:28.275007 17076 raft_consensus.cc:2454] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1333f5afd5b547e9ad5fa2782cbb38f3 in term 1.
I20260619 04:48:28.275075 17353 raft_consensus.cc:2454] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1333f5afd5b547e9ad5fa2782cbb38f3 in term 1.
I20260619 04:48:28.275398 17152 leader_election.cc:302] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1333f5afd5b547e9ad5fa2782cbb38f3, 66747ca451ff4fffb10b0df7b20e71e3; no voters:
I20260619 04:48:28.275780 17430 raft_consensus.cc:2781] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:48:28.276372 17430 raft_consensus.cc:686] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 1 LEADER]: Becoming Leader. State: Replica: 1333f5afd5b547e9ad5fa2782cbb38f3, State: Running, Role: LEADER
I20260619 04:48:28.276890 17430 consensus_queue.cc:237] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } }
I20260619 04:48:28.287408 16924 catalog_manager.cc:5495] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1333f5afd5b547e9ad5fa2782cbb38f3 (127.3.243.130). New cstate: current_term: 1 leader_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1333f5afd5b547e9ad5fa2782cbb38f3" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 39085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 } health_report { overall_health: UNKNOWN } } }
W20260619 04:48:28.327566 17122 tablet.cc:2369] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:48:28.350607 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.345150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.3.243.130:39085)
0619 04:48:28.345370 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:28.345376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:28.345396 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:48:28.345636 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:28.345640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:28.345734 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:28.345892 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:48:28.346250 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.346963 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.348402 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.348853 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.348986 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.349501 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:48:28.350011 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:48:28.350167 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:28.350228 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 04:48:28.350306 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:48:28.358712 17353 raft_consensus.cc:1261] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [term 1 FOLLOWER]: Refusing update from remote peer 1333f5afd5b547e9ad5fa2782cbb38f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:48:28.358842 17076 raft_consensus.cc:1261] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3 [term 1 FOLLOWER]: Refusing update from remote peer 1333f5afd5b547e9ad5fa2782cbb38f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:48:28.359817 17433 consensus_queue.cc:1035] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "749f8975157d4c6280cdc30ee93a801d" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:48:28.360253 17430 consensus_queue.cc:1035] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66747ca451ff4fffb10b0df7b20e71e3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 41793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:48:28.379962 17443 mvcc.cc:205] Tried to move back new op lower bound from 7298435106227720192 to 7298435105906925568. Current Snapshot: MvccSnapshot[applied={T|T < 7298435106227720192}]
I20260619 04:48:28.403975 16923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:48:28.404394 16923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:48:28.408098 16923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f36732cf3aa844818246f1e328af8c0c: Sending DeleteTablet for 3 replicas of tablet 070a3ff40f5f4c0393c0441c3cf77a47
I20260619 04:48:28.409304 17200 tablet_service.cc:1514] Processing DeleteTablet for tablet 070a3ff40f5f4c0393c0441c3cf77a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:48:28 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:51196
I20260619 04:48:28.409484 17056 tablet_service.cc:1514] Processing DeleteTablet for tablet 070a3ff40f5f4c0393c0441c3cf77a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:48:28 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:55986
I20260619 04:48:28.410360 17334 tablet_service.cc:1514] Processing DeleteTablet for tablet 070a3ff40f5f4c0393c0441c3cf77a47 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:48:28 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:48540
I20260619 04:48:28.421013 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 16961
I20260619 04:48:28.423671 17452 tablet_replica.cc:330] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: stopping tablet replica
I20260619 04:48:28.426221 17452 raft_consensus.cc:2229] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:48:28.427116 17452 raft_consensus.cc:2258] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:48:28.431556 17452 ts_tablet_manager.cc:1905] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:48:28.440480 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.422176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.3.243.129:41793)
0619 04:48:28.422365 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:28.422371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:28.422390 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:48:28.422449 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:28.422455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:28.422537 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:28.422652 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:28.424493 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.425744 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.427734 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.428464 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.428625 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.432315 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:28.432334 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:28.432355 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:28.432407 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:28.436983 (+ 4576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:28.437771 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:28.437782 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:28.437788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:28.437985 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:28.438428 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:28.438435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:28.438441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:28.438942 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:28.439340 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:28.439590 (+ 250us) server_negotiation.cc:299] Negotiation successful
0619 04:48:28.439867 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 04:48:28.441540 17116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:28.447189 17454 tablet_replica.cc:330] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: stopping tablet replica
I20260619 04:48:28.447741 17454 raft_consensus.cc:2229] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:48:28.448351 17454 raft_consensus.cc:2258] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:48:28.450325 17454 ts_tablet_manager.cc:1905] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:48:28.460749 17454 ts_tablet_manager.cc:1918] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:48:28.460924 17454 log.cc:1196] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/070a3ff40f5f4c0393c0441c3cf77a47
I20260619 04:48:28.461391 17454 ts_tablet_manager.cc:1939] T 070a3ff40f5f4c0393c0441c3cf77a47 P 749f8975157d4c6280cdc30ee93a801d: Deleting consensus metadata
I20260619 04:48:28.463100 17452 ts_tablet_manager.cc:1918] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:48:28.463280 17452 log.cc:1196] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/070a3ff40f5f4c0393c0441c3cf77a47
I20260619 04:48:28.463244 16908 catalog_manager.cc:4841] TS 749f8975157d4c6280cdc30ee93a801d (127.3.243.131:41911): tablet 070a3ff40f5f4c0393c0441c3cf77a47 (table test-table [id=3f2786a0802c4078a9375f47884223d1]) successfully deleted
I20260619 04:48:28.463761 17452 ts_tablet_manager.cc:1939] T 070a3ff40f5f4c0393c0441c3cf77a47 P 1333f5afd5b547e9ad5fa2782cbb38f3: Deleting consensus metadata
I20260619 04:48:28.465517 16910 catalog_manager.cc:4841] TS 1333f5afd5b547e9ad5fa2782cbb38f3 (127.3.243.130:39085): tablet 070a3ff40f5f4c0393c0441c3cf77a47 (table test-table [id=3f2786a0802c4078a9375f47884223d1]) successfully deleted
I20260619 04:48:28.613803 17453 tablet_replica.cc:330] T 070a3ff40f5f4c0393c0441c3cf77a47 P 66747ca451ff4fffb10b0df7b20e71e3: stopping tablet replica
I20260619 04:48:28.613874 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 16961
W20260619 04:48:28.630550 16909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.129:41793 (error 108)
I20260619 04:48:28.631870 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 17125
I20260619 04:48:28.632766 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.631834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.0.0.1:56002)
0619 04:48:28.632403 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.632547 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":321,"threads_started":1}
W20260619 04:48:28.633111 16909 catalog_manager.cc:4584] TS 66747ca451ff4fffb10b0df7b20e71e3 (127.3.243.129:41793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 070a3ff40f5f4c0393c0441c3cf77a47: Network error: Client connection negotiation failed: client connection to 127.3.243.129:41793: connect: Connection refused (error 111)
I20260619 04:48:28.652848 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.632660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.3.243.130:39085)
0619 04:48:28.632880 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:28.632886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:28.632947 (+ 61us) server_negotiation.cc:407] Connection header received
0619 04:48:28.633145 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:28.633150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:28.633232 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:28.633359 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:28.635021 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.635846 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.637699 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.638121 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.638210 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.646776 (+ 8566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:28.646789 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:28.646804 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:28.646845 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:28.649875 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:28.650621 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:28.650627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:28.650630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:28.650817 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:28.651200 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:28.651206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:28.651209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:28.651602 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:28.651934 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:28.652244 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 04:48:28.652479 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:48:28.653704 17260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:28.686946 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.686408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.0.0.1:56004)
0619 04:48:28.686651 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.686788 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 04:48:28.688069 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.687663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.0.0.1:56008)
0619 04:48:28.687866 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.687939 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 04:48:28.748541 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.748062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.0.0.1:56010)
0619 04:48:28.748307 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.748393 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 04:48:28.749593 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.749227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.0.0.1:56022)
0619 04:48:28.749394 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.749459 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 04:48:28.767443 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 17125
I20260619 04:48:28.785595 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 17269
I20260619 04:48:28.797904 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.786368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.3.243.131:41911)
0619 04:48:28.786765 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:28.786769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:28.786784 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:48:28.786836 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:28.786840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:28.786895 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:28.786980 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:28.788197 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.788866 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.789981 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:28.790322 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:28.790418 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:28.792765 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:28.792777 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:28.792791 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:28.792819 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:28.795487 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:28.796197 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:28.796204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:28.796206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:28.796342 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:28.796645 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:28.796647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:28.796649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:28.796979 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:28.797200 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:28.797389 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 04:48:28.797545 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":207,"threads_started":1}
I20260619 04:48:28.798547 17394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:28.830137 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.829700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.0.0.1:56034)
0619 04:48:28.829916 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.829999 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 04:48:28.831212 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:28.830859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.0.0.1:56046)
0619 04:48:28.831030 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:28.831094 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 04:48:28.886808 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 17269
I20260619 04:48:28.902974 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 16891
I20260619 04:48:28.904227 16954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:29.015049 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:29.014430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:41793 (local address 127.0.0.1:56058)
0619 04:48:29.014691 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:29.014814 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:41793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 04:48:29.015229 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 16891
I20260619 04:48:29.039659 4046 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 04:48:29.375993 4046 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 04:48:29.376294 4046 mini_ranger.cc:66] Stopping Ranger...
I20260619 04:48:29.721535 4046 mini_ranger.cc:68] Stopped Ranger
2026-06-19 04:48:29.721 UTC [15943] LOG: received smart shutdown request
2026-06-19 04:48:29.724 UTC [15943] LOG: background worker "logical replication launcher" (PID 15952) exited with exit code 1
2026-06-19 04:48:29.725 UTC [15947] LOG: shutting down
2026-06-19 04:48:29.885 UTC [15943] LOG: database system is shut down
2026-06-19T04:48:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52104 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:48:30 dist-test-slave-2qp8 krb5kdc[17470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:48:30 dist-test-slave-2qp8 krb5kdc[17470](info): set up 2 sockets
Jun 19 04:48:30 dist-test-slave-2qp8 krb5kdc[17470](info): commencing operation
krb5kdc: starting...
W20260619 04:48:31.104806 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:48:31 dist-test-slave-2qp8 krb5kdc[17470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:48:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:48:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:48:31.340590 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46385
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:46385,127.3.243.189:42131,127.3.243.188:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:31.666873 17483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:31.667163 17483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:31.667331 17483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:31.674437 17483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:48:31.674536 17483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:31.674634 17483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:31.674713 17483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:48:31.674801 17483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:48:31.684590 17483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:46385,127.3.243.189:42131,127.3.243.188:34989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:31.685884 17483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:31.687179 17483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:31.698988 17488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:31.699082 17489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:31.699689 17491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:31.700444 17483 server_base.cc:1034] running on GCE node
I20260619 04:48:31.701342 17483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:31.702708 17483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:31.703949 17483 hybrid_clock.cc:648] HybridClock initialized: now 1781844511703914 us; error 48 us; skew 500 ppm
I20260619 04:48:31.704347 17483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:31 dist-test-slave-2qp8 krb5kdc[17470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844511, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:31.707432 17483 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:48:31.708879 17483 webserver.cc:460] Webserver started at http://127.3.243.190:32949/ using document root <none> and password file <none>
I20260619 04:48:31.709518 17483 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:31.709609 17483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:31.709846 17483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:48:31.712780 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "839b2ac9f9494d1cbe145b97a675c1aa"
format_stamp: "Formatted at 2026-06-19 04:48:31 on dist-test-slave-2qp8"
server_key: "82447074be3cdf18e13fb85de848fde3"
server_key_iv: "d594b8c909512e09a2d9e8269f186be1"
server_key_version: "encryptionkey@0"
I20260619 04:48:31.713508 17483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "839b2ac9f9494d1cbe145b97a675c1aa"
format_stamp: "Formatted at 2026-06-19 04:48:31 on dist-test-slave-2qp8"
server_key: "82447074be3cdf18e13fb85de848fde3"
server_key_iv: "d594b8c909512e09a2d9e8269f186be1"
server_key_version: "encryptionkey@0"
I20260619 04:48:31.718729 17483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:48:31.722026 17498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:31.723301 17483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:48:31.723482 17483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "839b2ac9f9494d1cbe145b97a675c1aa"
format_stamp: "Formatted at 2026-06-19 04:48:31 on dist-test-slave-2qp8"
server_key: "82447074be3cdf18e13fb85de848fde3"
server_key_iv: "d594b8c909512e09a2d9e8269f186be1"
server_key_version: "encryptionkey@0"
I20260619 04:48:31.723672 17483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:31.743168 17483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:31.747068 17483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:31.747287 17483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:31.764196 17483 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:46385
I20260619 04:48:31.764259 17549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:46385 every 8 connection(s)
I20260619 04:48:31.765527 17483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:48:31.769093 17550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:31.772266 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 17483
I20260619 04:48:31.772480 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:48:31.773305 4046 external_mini_cluster.cc:1428] Setting key a86e5a5e9416f532cb159277c262d7c9
I20260619 04:48:31.777760 17550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:48:31 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844511, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.189@KRBTEST.COM" created.
I20260619 04:48:31.801077 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:31.781711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:46385 (local address 127.0.0.1:36252)
0619 04:48:31.782687 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:31.782736 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:31.784688 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:31.785477 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:31.785490 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:31.786170 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:31.786501 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:31.786518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:31.788064 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:31.788073 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:31.789188 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:31.789199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:31.790108 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:31.790115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:31.790321 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:31.791027 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:31.791058 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:31.793737 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:31.797096 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:31.797106 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:31.797114 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:31.797680 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:31.798141 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:31.798144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:31.798147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:31.798284 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:31.799255 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:31.799260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:31.799434 (+ 174us) client_negotiation.cc:769] Sending connection context
0619 04:48:31.799622 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 04:48:31.799980 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":435,"threads_started":1}
I20260619 04:48:31.801105 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:31.781711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.3.243.190:46385)
0619 04:48:31.783213 (+ 1502us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:31.783220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:31.783270 (+ 50us) server_negotiation.cc:407] Connection header received
0619 04:48:31.785051 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:31.785057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:31.785261 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:31.785464 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:31.786691 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:31.787902 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:31.789329 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:31.789881 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:31.790078 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:31.793953 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:31.793960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:31.793971 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:31.794001 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:31.796972 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:31.797824 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:31.797833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:31.797836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:31.798033 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:31.798402 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:31.798411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:31.798417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:31.799123 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:31.799325 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:31.799810 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 04:48:31.800218 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":1312,"spinlock_wait_cycles":44928,"thread_start_us":1102,"threads_started":1}
I20260619 04:48:31.804225 17550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:48:31.805562 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:31.805009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60446)
0619 04:48:31.805161 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:31.805411 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 04:48:31.805903 17499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
I20260619 04:48:31.807456 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:31.807110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60462)
0619 04:48:31.807263 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:31.807331 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 04:48:31.807677 17550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:42131: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab.
Entry for principal kudu/127.3.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab.
I20260619 04:48:31.825627 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.189:42131
--webserver_interface=127.3.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab
--principal=kudu/127.3.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:46385,127.3.243.189:42131,127.3.243.188:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:48:31.856968 17550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } attempt: 1
I20260619 04:48:31.858181 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:31.857774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60466)
0619 04:48:31.857954 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:31.858056 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 04:48:31.859092 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:31.858776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60470)
0619 04:48:31.858917 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:31.858980 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 04:48:31.859258 17550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:42131: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
I20260619 04:48:31.927513 17550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } attempt: 2
I20260619 04:48:31.928711 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:31.928224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60476)
0619 04:48:31.928492 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:31.928578 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 04:48:31.929682 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:31.929335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60480)
0619 04:48:31.929476 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:31.929557 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 04:48:31.929855 17550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:42131: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
I20260619 04:48:32.021136 17550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } attempt: 3
I20260619 04:48:32.022413 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.021968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60482)
0619 04:48:32.022192 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.022285 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260619 04:48:32.023545 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.023147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60490)
0619 04:48:32.023298 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.023398 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 04:48:32.023713 17550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:42131: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
W20260619 04:48:32.155831 17557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:32.156196 17557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:32.156379 17557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:32.163503 17557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:48:32.163653 17557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:32.163754 17557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:32.163863 17557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:48:32.163988 17557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:48:32.166949 17550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } attempt: 4
I20260619 04:48:32.168219 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.167643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60506)
0619 04:48:32.167949 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.168035 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260619 04:48:32.169210 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.168862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60522)
0619 04:48:32.169004 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.169097 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 04:48:32.169391 17550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:42131: Network error: Client connection negotiation failed: client connection to 127.3.243.189:42131: connect: Connection refused (error 111)
I20260619 04:48:32.175117 17557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:46385,127.3.243.189:42131,127.3.243.188:34989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab
--principal=kudu/127.3.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.189:42131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:32.176625 17557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:32.177968 17557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:32.199579 17563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:32.199622 17562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:32.199728 17565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:32.200999 17557 server_base.cc:1034] running on GCE node
I20260619 04:48:32.201874 17557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:32.203392 17557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:32.204655 17557 hybrid_clock.cc:648] HybridClock initialized: now 1781844512204607 us; error 57 us; skew 500 ppm
I20260619 04:48:32.205024 17557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:32.208034 17557 init.cc:377] Logged in from keytab as kudu/127.3.243.189@KRBTEST.COM (short username kudu)
I20260619 04:48:32.209635 17557 webserver.cc:460] Webserver started at http://127.3.243.189:36221/ using document root <none> and password file <none>
I20260619 04:48:32.210325 17557 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:32.210461 17557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:32.210754 17557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:48:32.213703 17557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/data/instance:
uuid: "4575b24ded4c4a2caad04e70ba4475fd"
format_stamp: "Formatted at 2026-06-19 04:48:32 on dist-test-slave-2qp8"
server_key: "c1db53508375e24838afa6fa0bdb5ed9"
server_key_iv: "414048f5ca45104f2146b3b76f6ebf07"
server_key_version: "encryptionkey@0"
I20260619 04:48:32.214494 17557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/wal/instance:
uuid: "4575b24ded4c4a2caad04e70ba4475fd"
format_stamp: "Formatted at 2026-06-19 04:48:32 on dist-test-slave-2qp8"
server_key: "c1db53508375e24838afa6fa0bdb5ed9"
server_key_iv: "414048f5ca45104f2146b3b76f6ebf07"
server_key_version: "encryptionkey@0"
I20260619 04:48:32.219460 17557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:48:32.222818 17572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:32.224090 17557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:48:32.224310 17557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/wal
uuid: "4575b24ded4c4a2caad04e70ba4475fd"
format_stamp: "Formatted at 2026-06-19 04:48:32 on dist-test-slave-2qp8"
server_key: "c1db53508375e24838afa6fa0bdb5ed9"
server_key_iv: "414048f5ca45104f2146b3b76f6ebf07"
server_key_version: "encryptionkey@0"
I20260619 04:48:32.224491 17557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:32.248597 17557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:32.252460 17557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:32.252722 17557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:32.272217 17557 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.189:42131
I20260619 04:48:32.272281 17623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.189:42131 every 8 connection(s)
I20260619 04:48:32.273629 17557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
I20260619 04:48:32.277279 17624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:32.282706 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 17557
I20260619 04:48:32.282905 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-1/wal/instance
I20260619 04:48:32.283351 4046 external_mini_cluster.cc:1428] Setting key ebf1797aa95fc86212858cd021f174f3
I20260619 04:48:32.283641 17624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.188@KRBTEST.COM; defaulting to no policy
I20260619 04:48:32.303143 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.286224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.3.243.190:46385)
0619 04:48:32.286446 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.286450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.287795 (+ 1345us) server_negotiation.cc:407] Connection header received
0619 04:48:32.289702 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.289708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.289802 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.289914 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.291365 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.292015 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.293481 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.293928 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.294056 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.297429 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.297434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.297437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.297469 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.300202 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.300922 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.300927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.300931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.301045 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.301509 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.301514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.301517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.301792 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.301956 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.302607 (+ 651us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.302824 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:48:32.304203 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.286857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:46385 (local address 127.0.0.1:36258)
0619 04:48:32.287621 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.287658 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:32.289427 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:32.290150 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:32.290159 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:32.290645 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:32.291182 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.291211 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.292196 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.292200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.293336 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.293347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.294148 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.294155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.294372 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.295035 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:32.295060 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:32.297202 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:32.300338 (+ 3136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.300346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.300362 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.300766 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.301186 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.301192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.301195 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.301391 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.301918 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:32.301927 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:32.302261 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 04:48:32.302514 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 04:48:32.303005 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":347,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.188@KRBTEST.COM" created.
I20260619 04:48:32.306200 17624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:48:32.321463 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.306998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60538)
0619 04:48:32.307158 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.307191 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:32.307295 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:32.308495 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:32.308500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:32.308942 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:32.309061 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.309069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.310445 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.310452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.311289 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.311299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.312136 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.312142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.312411 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.312992 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:32.313008 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:32.314965 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:32.318694 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.318701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.318707 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.319076 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.319494 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.319500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.319504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.319681 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.320396 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:32.320404 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:32.320556 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 04:48:32.320712 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 04:48:32.320992 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":39}
I20260619 04:48:32.321544 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.307295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.3.243.189:42131)
0619 04:48:32.308019 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.308025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.308050 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:48:32.308092 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.308097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.308349 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.308505 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.309232 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.310311 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.311426 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.311977 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.312110 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.315171 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.315181 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.315194 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.315221 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.318541 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.319227 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.319244 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.319246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.319361 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.319802 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.319810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.319818 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.320251 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.320477 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.320873 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.321085 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":569,"thread_start_us":429,"threads_started":1}
I20260619 04:48:32.324323 17624 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:48:32.325377 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.324903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57208)
0619 04:48:32.325131 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.325221 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 04:48:32.325678 17573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
I20260619 04:48:32.327368 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.326943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57212)
0619 04:48:32.327148 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.327250 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260619 04:48:32.327696 17624 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:34989: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab.
Entry for principal kudu/127.3.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab.
I20260619 04:48:32.331097 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.188:34989
--webserver_interface=127.3.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab
--principal=kudu/127.3.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:46385,127.3.243.189:42131,127.3.243.188:34989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:48:32.377050 17624 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } attempt: 1
I20260619 04:48:32.378319 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.377817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57218)
0619 04:48:32.378051 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.378158 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 04:48:32.379508 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.379124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57234)
0619 04:48:32.379334 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.379401 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260619 04:48:32.379845 17624 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:34989: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
I20260619 04:48:32.448091 17624 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } attempt: 2
I20260619 04:48:32.449286 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.448837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57240)
0619 04:48:32.449034 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.449130 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 04:48:32.450253 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.449926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57256)
0619 04:48:32.450069 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.450140 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 04:48:32.450464 17624 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:34989: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
I20260619 04:48:32.468664 17550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } attempt: 5
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844511, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:48:32.482630 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.469691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.3.243.189:42131)
0619 04:48:32.469898 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.469904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.469929 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:48:32.470034 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.470039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.470105 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.470240 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.471109 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.471852 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.473177 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.473631 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.473801 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.477321 (+ 3520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.477326 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.477329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.477362 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.479843 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.480689 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.480693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.480695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.480838 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.481294 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.481298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.481300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.481630 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.481759 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.482252 (+ 493us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.482418 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 04:48:32.482997 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.469506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60544)
0619 04:48:32.469706 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.469731 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:32.469879 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:32.470281 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:32.470287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:32.470726 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:32.470927 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.470937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.472000 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.472008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.472999 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.473013 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.473816 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.473822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.474026 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.474698 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:32.474715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:32.477133 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:32.479986 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.479991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.479995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.480538 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.480955 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.480960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.480963 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.481137 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.481756 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:32.481761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:32.481951 (+ 190us) client_negotiation.cc:769] Sending connection context
0619 04:48:32.482102 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 04:48:32.482388 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 04:48:32.484215 17550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:48:32.485277 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.484911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57260)
0619 04:48:32.485071 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.485146 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 04:48:32.486222 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.485907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57266)
0619 04:48:32.486056 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.486131 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 04:48:32.486445 17550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:34989: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
I20260619 04:48:32.537673 17550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } attempt: 1
I20260619 04:48:32.538851 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.538393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57274)
0619 04:48:32.538581 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.538674 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 04:48:32.539832 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.539491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57280)
0619 04:48:32.539654 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.539728 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 04:48:32.540031 17550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:34989: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
I20260619 04:48:32.541668 17624 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } attempt: 3
I20260619 04:48:32.542620 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.542214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57284)
0619 04:48:32.542396 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.542470 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 04:48:32.543591 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.543206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57298)
0619 04:48:32.543425 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.543490 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260619 04:48:32.543771 17624 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:34989: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
I20260619 04:48:32.608258 17550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } attempt: 2
I20260619 04:48:32.609325 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.608875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57306)
0619 04:48:32.609078 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.609168 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 04:48:32.610368 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.610042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57322)
0619 04:48:32.610192 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.610271 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 04:48:32.610572 17550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:34989: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
W20260619 04:48:32.656663 17631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:32.657006 17631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:32.657223 17631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:32.664239 17631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:48:32.664388 17631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:32.664501 17631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:32.664608 17631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:48:32.664677 17631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:48:32.674288 17631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:46385,127.3.243.189:42131,127.3.243.188:34989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab
--principal=kudu/127.3.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.188:34989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:32.675622 17631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:32.676930 17631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 04:48:32.686959 17624 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } attempt: 4
I20260619 04:48:32.688391 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.687649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57336)
0619 04:48:32.687946 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.688078 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 04:48:32.688362 17637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:32.688637 17636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:32.689076 17639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:32.689574 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.689089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57340)
0619 04:48:32.689287 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.689410 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 04:48:32.689805 17624 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:34989: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
I20260619 04:48:32.689853 17631 server_base.cc:1034] running on GCE node
I20260619 04:48:32.690650 17631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:32.691931 17631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:32.693161 17631 hybrid_clock.cc:648] HybridClock initialized: now 1781844512693104 us; error 47 us; skew 500 ppm
I20260619 04:48:32.693493 17631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:32.696580 17631 init.cc:377] Logged in from keytab as kudu/127.3.243.188@KRBTEST.COM (short username kudu)
I20260619 04:48:32.698040 17631 webserver.cc:460] Webserver started at http://127.3.243.188:43909/ using document root <none> and password file <none>
I20260619 04:48:32.698707 17631 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:32.698848 17631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:32.699146 17631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:48:32.702019 17631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/data/instance:
uuid: "1e155742071a4eadb3d3110988e5486d"
format_stamp: "Formatted at 2026-06-19 04:48:32 on dist-test-slave-2qp8"
server_key: "af36376672af3f8572d074172fee7960"
server_key_iv: "8cf821ade7d544a61a7dd975501f6b50"
server_key_version: "encryptionkey@0"
I20260619 04:48:32.702824 17631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/wal/instance:
uuid: "1e155742071a4eadb3d3110988e5486d"
format_stamp: "Formatted at 2026-06-19 04:48:32 on dist-test-slave-2qp8"
server_key: "af36376672af3f8572d074172fee7960"
server_key_iv: "8cf821ade7d544a61a7dd975501f6b50"
server_key_version: "encryptionkey@0"
I20260619 04:48:32.707796 17631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 04:48:32.711195 17646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:32.712554 17631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 04:48:32.712736 17631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/wal
uuid: "1e155742071a4eadb3d3110988e5486d"
format_stamp: "Formatted at 2026-06-19 04:48:32 on dist-test-slave-2qp8"
server_key: "af36376672af3f8572d074172fee7960"
server_key_iv: "8cf821ade7d544a61a7dd975501f6b50"
server_key_version: "encryptionkey@0"
I20260619 04:48:32.712932 17631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:32.716775 17550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } attempt: 3
I20260619 04:48:32.718048 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.717408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57352)
0619 04:48:32.717719 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.717837 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260619 04:48:32.719128 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.718702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57368)
0619 04:48:32.718900 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.718990 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 04:48:32.719365 17550 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:34989: Network error: Client connection negotiation failed: client connection to 127.3.243.188:34989: connect: Connection refused (error 111)
I20260619 04:48:32.729050 17631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:32.733338 17631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:32.733564 17631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:32.751547 17631 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.188:34989
I20260619 04:48:32.751570 17697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.188:34989 every 8 connection(s)
I20260619 04:48:32.752938 17631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
I20260619 04:48:32.756664 17698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:32.762087 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 17631
I20260619 04:48:32.762264 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781844312210132-4046-0/minicluster-data/master-2/wal/instance
I20260619 04:48:32.762686 4046 external_mini_cluster.cc:1428] Setting key 851c1d4c588515af58fa5e3d05c4534a
I20260619 04:48:32.762830 17698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
I20260619 04:48:32.795464 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.766366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36268 (local address 127.3.243.190:46385)
0619 04:48:32.766606 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.766615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.777124 (+ 10509us) server_negotiation.cc:407] Connection header received
0619 04:48:32.780182 (+ 3058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.780187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.780278 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.780397 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.782013 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.782702 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.784409 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.784835 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.784970 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.788712 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.788717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.788720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.788754 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.791506 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.792413 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.792418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.792422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.792551 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.792974 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.792978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.792982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.793788 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.793963 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.794868 (+ 905us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.795106 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:48:32.796674 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.765831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:46385 (local address 127.0.0.1:36268)
0619 04:48:32.776894 (+ 11063us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.776956 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:32.779844 (+ 2888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:32.780627 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:32.780636 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:32.781189 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:32.781835 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.781853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.782896 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.782909 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.784266 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.784277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.785064 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.785071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.785295 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.785999 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:32.786022 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:32.788526 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:32.791646 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.791657 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.791668 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.792288 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.792659 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.792665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.792668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.792860 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.793947 (+ 1087us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:32.793957 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:32.794433 (+ 476us) client_negotiation.cc:769] Sending connection context
0619 04:48:32.794737 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 04:48:32.795207 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10700,"mutex_wait_us":21,"thread_start_us":11320,"threads_started":1}
I20260619 04:48:32.796967 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.764482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.3.243.188:34989)
0619 04:48:32.778563 (+ 14081us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.778576 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.780752 (+ 2176us) server_negotiation.cc:407] Connection header received
0619 04:48:32.780976 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.780984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.781171 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.781363 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.782775 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.784224 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.785537 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.786082 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.786226 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.790689 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.790696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.790708 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.790736 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.793872 (+ 3136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.794745 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.794752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.794755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.794930 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.795359 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.795364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.795367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.795801 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.796031 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.796382 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.796555 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":13883,"thread_start_us":13653,"threads_started":1}
I20260619 04:48:32.798514 17698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:32.811600 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.799188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:42131 (local address 127.0.0.1:60552)
0619 04:48:32.799380 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.799400 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:32.799485 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:32.799810 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:32.799813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:32.800261 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:32.800432 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.800440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.801821 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.801825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.802663 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.802672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.803370 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.803376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.803579 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.804175 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:32.804201 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:32.806253 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:32.809351 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.809355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.809358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.809672 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.810053 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.810058 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.810061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.810189 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.810664 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:32.810668 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:32.810837 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 04:48:32.810967 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 04:48:32.811212 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 04:48:32.811625 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.799330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.3.243.189:42131)
0619 04:48:32.799498 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.799502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.799518 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:48:32.799596 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.799601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.799676 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.799779 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.800570 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.801686 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.802824 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.803228 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.803365 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.806446 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.806451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.806454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.806485 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.809182 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.809796 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.809803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.809806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.809947 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.810296 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.810301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.810305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.810551 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.810695 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.811062 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.811280 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:48:32.812764 17698 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:48:32.814134 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.800240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.3.243.190:46385)
0619 04:48:32.800401 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.800408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.800425 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:48:32.800594 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.800598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.800650 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.800714 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.802142 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.802865 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.804147 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.804564 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.804692 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.808624 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.808635 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.808638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.808667 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.811268 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.812078 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.812083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.812086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.812274 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.812670 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.812675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.812679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.812972 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.813225 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.813595 (+ 370us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.813840 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:48:32.826197 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.813507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57388)
0619 04:48:32.813680 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.813698 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:32.813826 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:32.814160 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:32.814164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:32.814512 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:32.814620 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.814627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.815730 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.815738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.817203 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.817219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.817942 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.817946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.818091 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.818603 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:32.818617 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:32.820817 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:32.823690 (+ 2873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.823694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.823696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.824078 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.824484 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.824487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.824489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.824606 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.825223 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:32.825229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:32.825396 (+ 167us) client_negotiation.cc:769] Sending connection context
0619 04:48:32.825537 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 04:48:32.825807 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 04:48:32.826454 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.813741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.3.243.188:34989)
0619 04:48:32.813935 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.813940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.813956 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:48:32.813997 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.814001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.814066 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.814146 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.814743 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.815590 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.817343 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.817787 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.817922 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.820986 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.820991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.820994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.821019 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.823556 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.824231 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.824236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.824239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.824378 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.824725 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.824740 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.824743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.825085 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.825254 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.825632 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.825968 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":59}
I20260619 04:48:32.831094 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.815706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.3.243.189:42131)
0619 04:48:32.815925 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.815934 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.815972 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:48:32.816183 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.816188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.816281 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.816394 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.817883 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.818775 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.820490 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.820875 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.821003 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.825275 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.825281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.825285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.825331 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.828714 (+ 3383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.829346 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.829351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.829355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.829502 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.829828 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.829833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.829836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.830138 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.830402 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.830560 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.830796 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:48:32.832207 17698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d: Bootstrap starting.
I20260619 04:48:32.835892 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:48:32.835886 17698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:32.837147 17698 log.cc:824] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:32.840402 17698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d: No bootstrap required, opened a new log
I20260619 04:48:32.851730 17698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } }
I20260619 04:48:32.852926 17698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:32.853096 17698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e155742071a4eadb3d3110988e5486d, State: Initialized, Role: FOLLOWER
I20260619 04:48:32.853806 17698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } }
I20260619 04:48:32.860280 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.841781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.3.243.190:46385)
0619 04:48:32.841970 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.841975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.842637 (+ 662us) server_negotiation.cc:407] Connection header received
0619 04:48:32.842898 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.842906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.842995 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.843066 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.845432 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.846505 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.848642 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.849289 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.849395 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.853463 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.853469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.853473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.853509 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.857038 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.857916 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.857922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.857926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.858078 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.858573 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.858578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.858581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.858926 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.859137 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.859667 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.859893 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:48:32.863571 17711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:32.864133 17711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:48:32.865108 17698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:48:32.870136 17628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.841944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.3.243.189:42131)
0619 04:48:32.842109 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.842115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.852844 (+ 10729us) server_negotiation.cc:407] Connection header received
0619 04:48:32.853178 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.853185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.853264 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.853338 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.855559 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.856785 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.858875 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.859490 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.859581 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.863146 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.863153 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.863161 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.863209 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.866616 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.867430 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.867437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.867443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.867608 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.868101 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.868135 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.868139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.868545 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.868840 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.869490 (+ 650us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.869765 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:48:32.873027 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.842982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57402 (local address 127.3.243.188:34989)
0619 04:48:32.843203 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.843211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.853246 (+ 10035us) server_negotiation.cc:407] Connection header received
0619 04:48:32.853592 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.853597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.853686 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.853750 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.856072 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.857511 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.860322 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.861225 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.861351 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.865283 (+ 3932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.865304 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.865312 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.865354 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.869339 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.870193 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.870197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.870199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.870486 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.870939 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.870944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.870946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.871321 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.871551 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.872456 (+ 905us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.872662 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 04:48:32.876533 17722 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:48:32.876685 17722 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:48:32.872812 17631 master_runner.cc:186] Error getting master registration for 127.3.243.190:46385: OK, instance_id { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" instance_seqno: 1781844511759302 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:48:32.881546 17631 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:48:32.896598 17550 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } attempt: 4
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844511, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
I20260619 04:48:32.912187 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.897389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.3.243.188:34989)
0619 04:48:32.897545 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.897549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.897564 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:48:32.897603 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.897607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.897660 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.897733 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.898730 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.899532 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.901510 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.902213 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.902389 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.905541 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.905548 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.905554 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.905601 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:32.909056 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.909890 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.909895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.909899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.910448 (+ 549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:32.910859 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:32.910867 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:32.910873 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:32.911184 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:32.911290 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:32.911712 (+ 422us) server_negotiation.cc:299] Negotiation successful
0619 04:48:32.911872 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:48:32.912318 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.897165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57414)
0619 04:48:32.897374 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.897394 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:32.897507 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:32.897794 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:32.897799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:32.898345 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:32.898595 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.898606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.899666 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.899672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.901130 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.901146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.902397 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.902401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.902542 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.903153 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:32.903171 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:32.905366 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:32.909181 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.909185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.909188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.909748 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.910557 (+ 809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:32.910564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:32.910569 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:32.910733 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:32.911299 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:32.911304 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:32.911479 (+ 175us) client_negotiation.cc:769] Sending connection context
0619 04:48:32.911632 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 04:48:32.911904 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 04:48:32.918606 17550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa: Bootstrap starting.
I20260619 04:48:32.922706 17550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:32.924043 17550 log.cc:824] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:32.927277 17550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa: No bootstrap required, opened a new log
I20260619 04:48:32.935998 17550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } }
I20260619 04:48:32.936403 17550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:32.936542 17550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839b2ac9f9494d1cbe145b97a675c1aa, State: Initialized, Role: FOLLOWER
I20260619 04:48:32.937161 17550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } }
I20260619 04:48:32.938133 17725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:32.938522 17725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:48:32.939316 17550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [sys.catalog]: configured and running, proceeding with master startup.
W20260619 04:48:32.950469 17736 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:48:32.950713 17736 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:48:32.948611 17483 master_runner.cc:186] Error getting master registration for 127.3.243.189:42131: OK, instance_id { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" instance_seqno: 1781844512266521 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:48:32.960943 17483 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 04:48:32.985653 17652 tablet.cc:2369] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:48:32.989034 17624 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } attempt: 5
Jun 19 04:48:32 dist-test-slave-2qp8 krb5kdc[17470](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844512, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
W20260619 04:48:33.000032 17504 tablet.cc:2369] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:48:33.003393 17711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:33.003700 17711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } }
I20260619 04:48:33.006486 17525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e155742071a4eadb3d3110988e5486d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" is_pre_election: true
I20260619 04:48:33.007069 17525 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e155742071a4eadb3d3110988e5486d in term 0.
I20260619 04:48:33.008605 17648 leader_election.cc:302] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e155742071a4eadb3d3110988e5486d, 839b2ac9f9494d1cbe145b97a675c1aa; no voters:
I20260619 04:48:33.008912 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.989633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:34989 (local address 127.0.0.1:57418)
0619 04:48:32.989879 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:32.989901 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:32.990024 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:32.990404 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:32.990409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:32.990944 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:32.991147 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.991160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.992612 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.992619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.994053 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:32.994068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.995101 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.995108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:32.995355 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.996045 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:32.996069 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:32.998805 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:33.005453 (+ 6648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:33.005460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:33.005465 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:33.006020 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:33.006465 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:33.006471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:33.006475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:33.006627 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:33.007345 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:33.007350 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:33.007509 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 04:48:33.007670 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 04:48:33.007968 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260619 04:48:33.005465 17711 leader_election.cc:288] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 839b2ac9f9494d1cbe145b97a675c1aa (127.3.243.190:46385), 4575b24ded4c4a2caad04e70ba4475fd (127.3.243.189:42131)
I20260619 04:48:33.009456 17701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:32.989691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.3.243.188:34989)
0619 04:48:32.989873 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:32.989880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:32.989979 (+ 99us) server_negotiation.cc:407] Connection header received
0619 04:48:32.990146 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:32.990152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:32.990227 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:32.990347 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:32.991300 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.992459 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.994203 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:32.994945 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:32.995071 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:32.999002 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:32.999009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:32.999015 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:32.999059 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:33.005297 (+ 6238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:33.006163 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:33.006171 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:33.006179 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:33.006332 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:33.006763 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:33.006768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:33.006772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:33.007143 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:33.007283 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:33.008288 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0619 04:48:33.008527 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:48:33.009516 17711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:48:33.009673 17711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:48:33.009768 17711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:33.013005 17711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } }
I20260619 04:48:33.014164 17525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e155742071a4eadb3d3110988e5486d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "839b2ac9f9494d1cbe145b97a675c1aa"
I20260619 04:48:33.014385 17525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:33.015020 17624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd: Bootstrap starting.
I20260619 04:48:33.017709 17525 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e155742071a4eadb3d3110988e5486d in term 1.
I20260619 04:48:33.018239 17648 leader_election.cc:302] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e155742071a4eadb3d3110988e5486d, 839b2ac9f9494d1cbe145b97a675c1aa; no voters:
I20260619 04:48:33.018636 17738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:48:33.019280 17624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:33.020637 17624 log.cc:824] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:33.024593 17711 leader_election.cc:288] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [CANDIDATE]: Term 1 election: Requested vote from peers 839b2ac9f9494d1cbe145b97a675c1aa (127.3.243.190:46385), 4575b24ded4c4a2caad04e70ba4475fd (127.3.243.189:42131)
I20260619 04:48:33.033499 17738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [term 1 LEADER]: Becoming Leader. State: Replica: 1e155742071a4eadb3d3110988e5486d, State: Running, Role: LEADER
I20260619 04:48:33.034792 17738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } }
I20260619 04:48:33.035959 17624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd: No bootstrap required, opened a new log
I20260619 04:48:33.049561 17624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } }
I20260619 04:48:33.050040 17624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:33.050175 17624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4575b24ded4c4a2caad04e70ba4475fd, State: Initialized, Role: FOLLOWER
I20260619 04:48:33.050475 17711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e155742071a4eadb3d3110988e5486d. Latest consensus state: current_term: 1 leader_uuid: "1e155742071a4eadb3d3110988e5486d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:33.050772 17711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:33.050997 17624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } }
I20260619 04:48:33.058154 17743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:33.058549 17743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:48:33.060655 17598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e155742071a4eadb3d3110988e5486d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4575b24ded4c4a2caad04e70ba4475fd"
I20260619 04:48:33.061179 17598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:33.062832 17742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:48:33.064792 17598 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e155742071a4eadb3d3110988e5486d in term 1.
I20260619 04:48:33.065176 17742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:48:33.065650 17599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e155742071a4eadb3d3110988e5486d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4575b24ded4c4a2caad04e70ba4475fd" is_pre_election: true
I20260619 04:48:33.065981 17599 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 1e155742071a4eadb3d3110988e5486d in term 1. Re-sending same reply.
I20260619 04:48:33.066576 17624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:48:33.088418 17598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [term 1 FOLLOWER]: Refusing update from remote peer 1e155742071a4eadb3d3110988e5486d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:48:33.089588 17711 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:48:33.101922 17525 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [term 1 FOLLOWER]: Refusing update from remote peer 1e155742071a4eadb3d3110988e5486d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:48:33.102944 17711 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [LEADER]: Connected to new peer: Peer: permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:48:33.105873 17743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e155742071a4eadb3d3110988e5486d. Latest consensus state: current_term: 1 leader_uuid: "1e155742071a4eadb3d3110988e5486d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:33.106276 17743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:48:33.110090 17745 mvcc.cc:205] Tried to move back new op lower bound from 7298435125595172864 to 7298435125401387008. Current Snapshot: MvccSnapshot[applied={T|T < 7298435125595172864}]
I20260619 04:48:33.111258 17750 mvcc.cc:205] Tried to move back new op lower bound from 7298435125595172864 to 7298435125401387008. Current Snapshot: MvccSnapshot[applied={T|T < 7298435125595172864}]
I20260619 04:48:33.117472 17743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e155742071a4eadb3d3110988e5486d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:33.117448 17738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e155742071a4eadb3d3110988e5486d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:33.117745 17743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:48:33.117779 17738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:33.120513 17742 catalog_manager.cc:1347] Generated new cluster ID: af48ad4c00e04b428cf5bb2573a8c6c5
I20260619 04:48:33.120601 17742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:48:33.121323 17725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e155742071a4eadb3d3110988e5486d. Latest consensus state: current_term: 1 leader_uuid: "1e155742071a4eadb3d3110988e5486d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:33.121583 17725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:48:33.128952 17739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1e155742071a4eadb3d3110988e5486d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:33.129410 17739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:33.130373 17754 mvcc.cc:205] Tried to move back new op lower bound from 7298435125595172864 to 7298435125401387008. Current Snapshot: MvccSnapshot[applied={T|T < 7298435125595172864}]
I20260619 04:48:33.139698 17725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1e155742071a4eadb3d3110988e5486d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "839b2ac9f9494d1cbe145b97a675c1aa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46385 } } peers { permanent_uuid: "4575b24ded4c4a2caad04e70ba4475fd" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 42131 } } peers { permanent_uuid: "1e155742071a4eadb3d3110988e5486d" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 34989 } } }
I20260619 04:48:33.139997 17725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:48:33.144418 17742 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:48:33.146167 17742 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:48:33.162731 17742 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e155742071a4eadb3d3110988e5486d: Generated new TSK 0
I20260619 04:48:33.163961 17742 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:48:33.192071 17762 catalog_manager.cc:1259] Loaded cluster ID: af48ad4c00e04b428cf5bb2573a8c6c5
I20260619 04:48:33.192337 17762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd: loading cluster ID for follower catalog manager: success
I20260619 04:48:33.197405 17762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd: acquiring CA information for follower catalog manager: success
I20260619 04:48:33.201738 17762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4575b24ded4c4a2caad04e70ba4475fd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:48:33.952769 17736 catalog_manager.cc:1259] Loaded cluster ID: af48ad4c00e04b428cf5bb2573a8c6c5
I20260619 04:48:33.952867 17736 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa: loading cluster ID for follower catalog manager: success
I20260619 04:48:33.955251 17736 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa: acquiring CA information for follower catalog manager: success
I20260619 04:48:33.957046 17736 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 839b2ac9f9494d1cbe145b97a675c1aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:48:34.278280 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 17483
I20260619 04:48:34.279963 17545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:34.392263 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 17483
I20260619 04:48:34.414357 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 17557
I20260619 04:48:34.415439 17619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:34.522933 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 17557
I20260619 04:48:34.545424 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 17631
I20260619 04:48:34.546641 17693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:34.657997 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 17631
2026-06-19T04:48:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4643 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:48:34 dist-test-slave-2qp8 krb5kdc[17775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:48:34 dist-test-slave-2qp8 krb5kdc[17775](info): set up 2 sockets
Jun 19 04:48:34 dist-test-slave-2qp8 krb5kdc[17775](info): commencing operation
krb5kdc: starting...
W20260619 04:48:35.752969 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:48:35 dist-test-slave-2qp8 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:48:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:48:35.984007 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46503
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:46503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:36.311894 17788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:36.312316 17788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:36.312491 17788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:36.319705 17788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:48:36.319804 17788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:36.319900 17788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:36.320008 17788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:48:36.320091 17788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:48:36.329780 17788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:46503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:36.331139 17788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:36.332533 17788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:36.344192 17793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:36.344245 17794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:36.344427 17796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:36.345016 17788 server_base.cc:1034] running on GCE node
I20260619 04:48:36.345783 17788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:36.347077 17788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:36.348343 17788 hybrid_clock.cc:648] HybridClock initialized: now 1781844516348310 us; error 35 us; skew 500 ppm
I20260619 04:48:36.348634 17788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:36 dist-test-slave-2qp8 krb5kdc[17775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:36.351680 17788 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:48:36.353116 17788 webserver.cc:460] Webserver started at http://127.3.243.190:39265/ using document root <none> and password file <none>
I20260619 04:48:36.353821 17788 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:36.353931 17788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:36.354249 17788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:48:36.357154 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "3fd87fe3a64941f290911415a4ea421f"
format_stamp: "Formatted at 2026-06-19 04:48:36 on dist-test-slave-2qp8"
server_key: "e7195df82274de110eac97c00909d641"
server_key_iv: "95068240f2f0c8f29847335eb441c636"
server_key_version: "encryptionkey@0"
I20260619 04:48:36.357838 17788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "3fd87fe3a64941f290911415a4ea421f"
format_stamp: "Formatted at 2026-06-19 04:48:36 on dist-test-slave-2qp8"
server_key: "e7195df82274de110eac97c00909d641"
server_key_iv: "95068240f2f0c8f29847335eb441c636"
server_key_version: "encryptionkey@0"
I20260619 04:48:36.362737 17788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:48:36.366118 17803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:36.367424 17788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:48:36.367568 17788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "3fd87fe3a64941f290911415a4ea421f"
format_stamp: "Formatted at 2026-06-19 04:48:36 on dist-test-slave-2qp8"
server_key: "e7195df82274de110eac97c00909d641"
server_key_iv: "95068240f2f0c8f29847335eb441c636"
server_key_version: "encryptionkey@0"
I20260619 04:48:36.367725 17788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:36.385231 17788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:36.389580 17788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:36.389817 17788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:36.408305 17788 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:46503
I20260619 04:48:36.408315 17854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:46503 every 8 connection(s)
I20260619 04:48:36.409650 17788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:48:36.413355 17855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:36.415406 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 17788
I20260619 04:48:36.415613 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:48:36.416326 4046 external_mini_cluster.cc:1428] Setting key cd3377d2085ef43b2486bdea2323fc6b
I20260619 04:48:36.426661 17855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f: Bootstrap starting.
I20260619 04:48:36.430078 17855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:36.431077 17855 log.cc:824] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f: Log is configured to *not* fsync() on all Append() calls
Jun 19 04:48:36 dist-test-slave-2qp8 krb5kdc[17775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844515, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:36.434473 17855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f: No bootstrap required, opened a new log
I20260619 04:48:36.441361 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:36.418236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51154 (local address 127.3.243.190:46503)
0619 04:48:36.419051 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:36.419072 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:36.419121 (+ 49us) server_negotiation.cc:407] Connection header received
0619 04:48:36.421591 (+ 2470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:36.421619 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:36.422116 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:36.422541 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:36.423878 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:36.425891 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:36.427321 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:36.428407 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:36.428590 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:36.432937 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:36.432948 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:36.432966 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:36.433011 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:36.436952 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:36.437639 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:36.437647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:36.437662 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:36.437884 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:36.438217 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:36.438222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:36.438228 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:36.438918 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:36.439180 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:36.439490 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 04:48:36.439932 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":345,"threads_started":1}
I20260619 04:48:36.445302 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:48:36.445070 17855 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd87fe3a64941f290911415a4ea421f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46503 } }
I20260619 04:48:36.445458 17855 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:36.445560 17855 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fd87fe3a64941f290911415a4ea421f, State: Initialized, Role: FOLLOWER
I20260619 04:48:36.446149 17855 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd87fe3a64941f290911415a4ea421f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46503 } }
I20260619 04:48:36.446401 17855 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:48:36.446535 17855 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:48:36.446713 17855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:36.449240 17855 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd87fe3a64941f290911415a4ea421f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46503 } }
I20260619 04:48:36.449709 17855 leader_election.cc:302] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fd87fe3a64941f290911415a4ea421f; no voters:
I20260619 04:48:36.450321 17855 leader_election.cc:288] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:48:36.450456 17860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:48:36.451156 17860 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [term 1 LEADER]: Becoming Leader. State: Replica: 3fd87fe3a64941f290911415a4ea421f, State: Running, Role: LEADER
I20260619 04:48:36.451618 17860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd87fe3a64941f290911415a4ea421f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46503 } }
I20260619 04:48:36.452425 17855 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:48:36.454246 17862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fd87fe3a64941f290911415a4ea421f. Latest consensus state: current_term: 1 leader_uuid: "3fd87fe3a64941f290911415a4ea421f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd87fe3a64941f290911415a4ea421f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46503 } } }
I20260619 04:48:36.454589 17862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:36.456278 17866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:48:36.456066 17861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fd87fe3a64941f290911415a4ea421f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fd87fe3a64941f290911415a4ea421f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46503 } } }
I20260619 04:48:36.456394 17861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:36.461557 17866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:48:36.470788 17866 catalog_manager.cc:1347] Generated new cluster ID: 17b144de536c4fbb8bdc4579a589ad18
I20260619 04:48:36.470918 17866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:48:36.480024 17866 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:48:36.481359 17866 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:48:36.488163 17866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fd87fe3a64941f290911415a4ea421f: Generated new TSK 0
I20260619 04:48:36.488940 17866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:48:36.506026 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 17788
I20260619 04:48:36.507151 17850 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:36.600572 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 17788
2026-06-19T04:48:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1937 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:48:36 dist-test-slave-2qp8 krb5kdc[17883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:48:36 dist-test-slave-2qp8 krb5kdc[17883](info): set up 2 sockets
Jun 19 04:48:36 dist-test-slave-2qp8 krb5kdc[17883](info): commencing operation
krb5kdc: starting...
W20260619 04:48:37.684646 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:48:37 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:48:37.911895 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42719
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:42719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:38.232714 17896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:38.233062 17896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:38.233227 17896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:38.240264 17896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:48:38.240410 17896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:38.240522 17896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:38.240617 17896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:48:38.240703 17896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:48:38.250248 17896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:42719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:38.251582 17896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:38.252946 17896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:38.264823 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:38.264848 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:38.264775 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:38.265652 17896 server_base.cc:1034] running on GCE node
I20260619 04:48:38.266482 17896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:38.267913 17896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:38.269167 17896 hybrid_clock.cc:648] HybridClock initialized: now 1781844518269095 us; error 47 us; skew 500 ppm
I20260619 04:48:38.269552 17896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:38 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:38.272917 17896 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:48:38.274407 17896 webserver.cc:460] Webserver started at http://127.3.243.190:41973/ using document root <none> and password file <none>
I20260619 04:48:38.275089 17896 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:38.275202 17896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:38.275460 17896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:48:38.278570 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "999fee5983b04f05a72050eba740fada"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "2093314be382603aa216b7162f489c69"
server_key_iv: "53eb8741a19a7b21bf7acf737e126f4f"
server_key_version: "encryptionkey@0"
I20260619 04:48:38.279315 17896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "999fee5983b04f05a72050eba740fada"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "2093314be382603aa216b7162f489c69"
server_key_iv: "53eb8741a19a7b21bf7acf737e126f4f"
server_key_version: "encryptionkey@0"
I20260619 04:48:38.284732 17896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260619 04:48:38.288224 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:38.289507 17896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:48:38.289642 17896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "999fee5983b04f05a72050eba740fada"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "2093314be382603aa216b7162f489c69"
server_key_iv: "53eb8741a19a7b21bf7acf737e126f4f"
server_key_version: "encryptionkey@0"
I20260619 04:48:38.289860 17896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:38.334321 17896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:38.338299 17896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:38.338528 17896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:38.356777 17896 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:42719
I20260619 04:48:38.356807 17962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:42719 every 8 connection(s)
I20260619 04:48:38.358144 17896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:48:38.361725 17963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:38.363317 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 17896
I20260619 04:48:38.363531 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:48:38.364303 4046 external_mini_cluster.cc:1428] Setting key 0ab91b61c9a84a10883c9d3c0562b643
I20260619 04:48:38.373919 17963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Bootstrap starting.
I20260619 04:48:38.377743 17963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:38.379042 17963 log.cc:824] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Log is configured to *not* fsync() on all Append() calls
Jun 19 04:48:38 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:38.382203 17963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: No bootstrap required, opened a new log
I20260619 04:48:38.388531 17966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:38.366333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.3.243.190:42719)
0619 04:48:38.367109 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:38.367121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:38.367164 (+ 43us) server_negotiation.cc:407] Connection header received
0619 04:48:38.368926 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:38.368948 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:38.369346 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:38.369655 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:38.371499 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:38.372915 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:38.374737 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:38.375420 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:38.375551 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:38.380914 (+ 5363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:38.380924 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:38.380939 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:38.380977 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:38.384323 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:38.385071 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:38.385089 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:38.385095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:38.385262 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:38.385699 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:38.385704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:38.385717 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:38.386242 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:38.386612 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:38.387038 (+ 426us) server_negotiation.cc:299] Negotiation successful
0619 04:48:38.387477 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":346,"threads_started":1}
I20260619 04:48:38.390228 17963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:38.390620 17963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:38.390748 17963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999fee5983b04f05a72050eba740fada, State: Initialized, Role: FOLLOWER
I20260619 04:48:38.391386 17963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:38.391644 17963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:48:38.391788 17963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:48:38.391969 17963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:38.394518 17963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:38.395035 17963 leader_election.cc:302] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999fee5983b04f05a72050eba740fada; no voters:
I20260619 04:48:38.395574 17963 leader_election.cc:288] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:48:38.395694 17968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:48:38.396473 17968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 1 LEADER]: Becoming Leader. State: Replica: 999fee5983b04f05a72050eba740fada, State: Running, Role: LEADER
I20260619 04:48:38.397059 17968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:38.397755 17963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:48:38.399969 17970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999fee5983b04f05a72050eba740fada. Latest consensus state: current_term: 1 leader_uuid: "999fee5983b04f05a72050eba740fada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } } }
I20260619 04:48:38.400493 17970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:38.400024 17969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "999fee5983b04f05a72050eba740fada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } } }
I20260619 04:48:38.401015 17969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:38.401959 17976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:48:38.407061 17976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:48:38.417441 17976 catalog_manager.cc:1347] Generated new cluster ID: 5372fc6c60a843b9aaadf9d2304b265d
I20260619 04:48:38.417573 17976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:48:38.432286 17976 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:48:38.433578 17976 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:48:38.444366 17976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Generated new TSK 0
I20260619 04:48:38.445148 17976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:48:38.511271 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42719
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:38.840631 17989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:38.840917 17989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:38.841063 17989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:38.848519 17989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:38.848621 17989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:38.848826 17989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:48:38.858717 17989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:38.860154 17989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:38.861469 17989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:38.874780 17994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:38.874786 17997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:38.874836 17995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:38.875788 17989 server_base.cc:1034] running on GCE node
I20260619 04:48:38.876549 17989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:38.877537 17989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:38.878763 17989 hybrid_clock.cc:648] HybridClock initialized: now 1781844518878715 us; error 61 us; skew 500 ppm
I20260619 04:48:38.879107 17989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:38 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:38.882378 17989 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:48:38.883764 17989 webserver.cc:460] Webserver started at http://127.3.243.129:43021/ using document root <none> and password file <none>
I20260619 04:48:38.884552 17989 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:38.884683 17989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:38.884986 17989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:48:38.887820 17989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "8545b9f0426a4b84bd02876f772754d3"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "55cfbb922111ff19061733c7cc481457"
server_key_iv: "a0d529ce50316f1dbeeba8447af40a56"
server_key_version: "encryptionkey@0"
I20260619 04:48:38.888633 17989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "8545b9f0426a4b84bd02876f772754d3"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "55cfbb922111ff19061733c7cc481457"
server_key_iv: "a0d529ce50316f1dbeeba8447af40a56"
server_key_version: "encryptionkey@0"
I20260619 04:48:38.893745 17989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:48:38.897109 18004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:38.898522 17989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:48:38.898692 17989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "8545b9f0426a4b84bd02876f772754d3"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "55cfbb922111ff19061733c7cc481457"
server_key_iv: "a0d529ce50316f1dbeeba8447af40a56"
server_key_version: "encryptionkey@0"
I20260619 04:48:38.898931 17989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:38.916397 17989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:38.920092 17989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:38.920454 17989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:38.921412 17989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:38.922806 17989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:38.922874 17989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:38.923014 17989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:38.923070 17989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:38.955765 17989 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:33203
I20260619 04:48:38.955833 18116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:33203 every 8 connection(s)
I20260619 04:48:38.957079 17989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:48:38.963599 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 17989
I20260619 04:48:38.963835 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:48:38.964607 4046 external_mini_cluster.cc:1428] Setting key 7fe591b80b3bd5332c3d19ede6623e7d
Jun 19 04:48:38 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:38.978055 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:38.959608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:47275 (local address 127.3.243.190:42719)
0619 04:48:38.960015 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:38.960019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:38.961121 (+ 1102us) server_negotiation.cc:407] Connection header received
0619 04:48:38.962811 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:38.962816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:38.962881 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:38.962976 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:38.964611 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:38.965591 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:38.967483 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:38.968247 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:38.968404 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:38.971622 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:38.971631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:38.971639 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:38.971699 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:38.974769 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:38.975559 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:38.975567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:38.975573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:38.975714 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:38.976081 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:38.976089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:38.976095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:38.976517 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:38.976670 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:38.977317 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 04:48:38.977574 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":204,"threads_started":1}
I20260619 04:48:38.978690 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:38.960276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:47275)
0619 04:48:38.960961 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:38.961002 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:38.962565 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:38.963173 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:38.963181 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:38.963706 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:38.964438 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:38.964455 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:38.965776 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:38.965782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:38.967334 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:38.967346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:38.968456 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:38.968461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:38.968646 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:38.969322 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:38.969347 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:38.971426 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:38.974906 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:38.974918 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:38.974935 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:38.975446 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:38.975817 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:38.975820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:38.975822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:38.975981 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:38.976642 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:38.976647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:38.976955 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 04:48:38.977180 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 04:48:38.977568 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":257,"threads_started":1}
I20260619 04:48:38.980365 18117 heartbeater.cc:347] Connected to a master server at 127.3.243.190:42719
I20260619 04:48:38.980680 18117 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:38.981410 18117 heartbeater.cc:511] Master 127.3.243.190:42719 requested a full tablet report, sending...
I20260619 04:48:38.983492 17928 ts_manager.cc:194] Registered new tserver with Master: 8545b9f0426a4b84bd02876f772754d3 (127.3.243.129:33203)
I20260619 04:48:38.984965 17928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:47275
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:48:39.019340 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42719
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:39.356529 18123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:39.356892 18123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:39.357062 18123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:39.368521 18123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:39.368707 18123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:39.368906 18123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:48:39.382746 18123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:39.383988 18123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:39.385380 18123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:39.398720 18129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:39.398769 18128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:39.398793 18131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:39.400324 18123 server_base.cc:1034] running on GCE node
I20260619 04:48:39.400974 18123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:39.401943 18123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:39.403165 18123 hybrid_clock.cc:648] HybridClock initialized: now 1781844519403113 us; error 49 us; skew 500 ppm
I20260619 04:48:39.403584 18123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:39 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:39.406947 18123 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:48:39.408646 18123 webserver.cc:460] Webserver started at http://127.3.243.130:45059/ using document root <none> and password file <none>
I20260619 04:48:39.409340 18123 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:39.409492 18123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:39.409804 18123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:48:39.412808 18123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "9f1658cbafb44190845c63f6d446090c"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "431c5d0db143535e0e809daf19845b91"
server_key_iv: "256114625a1110da429b025be321fb20"
server_key_version: "encryptionkey@0"
I20260619 04:48:39.413558 18123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "9f1658cbafb44190845c63f6d446090c"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "431c5d0db143535e0e809daf19845b91"
server_key_iv: "256114625a1110da429b025be321fb20"
server_key_version: "encryptionkey@0"
I20260619 04:48:39.418620 18123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:48:39.422051 18138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:39.423434 18123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:48:39.423614 18123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "9f1658cbafb44190845c63f6d446090c"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "431c5d0db143535e0e809daf19845b91"
server_key_iv: "256114625a1110da429b025be321fb20"
server_key_version: "encryptionkey@0"
I20260619 04:48:39.423807 18123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:39.436275 18123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:39.440595 18123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:39.440872 18123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:39.441773 18123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:39.443058 18123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:39.443156 18123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:39.443297 18123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:39.443370 18123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:39.476341 18123 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:38681
I20260619 04:48:39.476423 18250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:38681 every 8 connection(s)
I20260619 04:48:39.477629 18123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:48:39.481639 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 18123
I20260619 04:48:39.481832 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:48:39.482363 4046 external_mini_cluster.cc:1428] Setting key 693677279b69797424aab78533ae71bb
Jun 19 04:48:39 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:39.501634 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:39.480359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:33027 (local address 127.3.243.190:42719)
0619 04:48:39.480814 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:39.480821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:39.481830 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 04:48:39.483857 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:39.483863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:39.483969 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:39.484101 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:39.486219 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:39.486950 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:39.489498 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:39.489911 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:39.490004 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:39.494364 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:39.494373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:39.494380 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:39.494424 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:39.498164 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:39.498923 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:39.498932 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:39.498938 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:39.499092 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:39.499606 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:39.499614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:39.499620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:39.500016 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:39.500256 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:39.500909 (+ 653us) server_negotiation.cc:299] Negotiation successful
0619 04:48:39.501165 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":199,"threads_started":1}
I20260619 04:48:39.502244 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:39.480985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:33027)
0619 04:48:39.481666 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:39.481705 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:39.483588 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:39.484425 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:39.484436 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:39.485175 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:39.486011 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:39.486039 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:39.487103 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:39.487111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:39.489362 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:39.489380 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:39.490131 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:39.490140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:39.490456 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:39.491178 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:39.491210 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:39.494119 (+ 2909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:39.498324 (+ 4205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:39.498336 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:39.498353 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:39.498801 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:39.499230 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:39.499237 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:39.499240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:39.499466 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:39.500218 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:39.500224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:39.500547 (+ 323us) client_negotiation.cc:769] Sending connection context
0619 04:48:39.500765 (+ 218us) client_negotiation.cc:240] Negotiation successful
0619 04:48:39.501135 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":285,"threads_started":1}
I20260619 04:48:39.503783 18251 heartbeater.cc:347] Connected to a master server at 127.3.243.190:42719
I20260619 04:48:39.504182 18251 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:39.504980 18251 heartbeater.cc:511] Master 127.3.243.190:42719 requested a full tablet report, sending...
I20260619 04:48:39.506491 17928 ts_manager.cc:194] Registered new tserver with Master: 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681)
I20260619 04:48:39.507313 17928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:33027
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:48:39.538787 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42719
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:39.857851 18257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:39.858161 18257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:39.858331 18257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:39.865298 18257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:39.865398 18257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:39.865504 18257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:48:39.874807 18257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:42719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:39.876044 18257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:39.877411 18257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:39.890444 18262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:39.890362 18263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:39.890399 18265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:39.891423 18257 server_base.cc:1034] running on GCE node
I20260619 04:48:39.891988 18257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:39.893034 18257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:39.894265 18257 hybrid_clock.cc:648] HybridClock initialized: now 1781844519894234 us; error 48 us; skew 500 ppm
I20260619 04:48:39.894582 18257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:39 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:39.902431 18257 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:48:39.903963 18257 webserver.cc:460] Webserver started at http://127.3.243.131:36537/ using document root <none> and password file <none>
I20260619 04:48:39.904738 18257 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:39.904881 18257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:39.905184 18257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:48:39.908152 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "01205c6127cc44bcbbffe4168838e233"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "a6db3955180c1edd91392a0870d277ee"
server_key_iv: "4fa463a5998b0ff67862c3558f9bff6e"
server_key_version: "encryptionkey@0"
I20260619 04:48:39.908936 18257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "01205c6127cc44bcbbffe4168838e233"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "a6db3955180c1edd91392a0870d277ee"
server_key_iv: "4fa463a5998b0ff67862c3558f9bff6e"
server_key_version: "encryptionkey@0"
I20260619 04:48:39.914361 18257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:48:39.917927 18272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:39.919198 18257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:48:39.919373 18257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "01205c6127cc44bcbbffe4168838e233"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "a6db3955180c1edd91392a0870d277ee"
server_key_iv: "4fa463a5998b0ff67862c3558f9bff6e"
server_key_version: "encryptionkey@0"
I20260619 04:48:39.919585 18257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:39.934046 18257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:39.938390 18257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:39.938681 18257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:39.939571 18257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:39.940917 18257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:39.941017 18257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:39.941169 18257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:39.941246 18257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:39.974727 18257 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:40083
I20260619 04:48:39.974763 18384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:40083 every 8 connection(s)
I20260619 04:48:39.976028 18257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:48:39.980015 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 18257
I20260619 04:48:39.980247 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:48:39.980791 4046 external_mini_cluster.cc:1428] Setting key 8cf1137f322634f7bb1300225af85dc4
I20260619 04:48:39.988013 18117 heartbeater.cc:503] Master 127.3.243.190:42719 was elected leader, sending a full tablet report...
Jun 19 04:48:39 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:39.997128 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:39.978697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:37223 (local address 127.3.243.190:42719)
0619 04:48:39.978892 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:39.978897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:39.980188 (+ 1291us) server_negotiation.cc:407] Connection header received
0619 04:48:39.982440 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:39.982444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:39.982521 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:39.982599 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:39.984407 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:39.985152 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:39.986621 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:39.987010 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:39.987111 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:39.990848 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:39.990853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:39.990856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:39.990885 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:39.993923 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:39.994781 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:39.994786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:39.994788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:39.994899 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:39.995392 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:39.995398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:39.995403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:39.995784 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:39.995958 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:39.996735 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 04:48:39.996898 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:48:39.998101 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:39.979146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:37223)
0619 04:48:39.980002 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:39.980059 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:39.982154 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:39.982830 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:39.982837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:39.983514 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:39.984267 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:39.984286 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:39.985309 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:39.985312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:39.986481 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:39.986491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:39.987188 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:39.987193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:39.987373 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:39.988038 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:39.988058 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:39.990682 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:39.994099 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:39.994111 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:39.994123 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:39.994647 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:39.995013 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:39.995019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:39.995022 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:39.995258 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:39.995927 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:39.995933 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:39.996369 (+ 436us) client_negotiation.cc:769] Sending connection context
0619 04:48:39.996616 (+ 247us) client_negotiation.cc:240] Negotiation successful
0619 04:48:39.996978 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":263,"threads_started":1}
I20260619 04:48:39.999475 18385 heartbeater.cc:347] Connected to a master server at 127.3.243.190:42719
I20260619 04:48:39.999804 18385 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:40.000490 18385 heartbeater.cc:511] Master 127.3.243.190:42719 requested a full tablet report, sending...
I20260619 04:48:40.001775 17928 ts_manager.cc:194] Registered new tserver with Master: 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
I20260619 04:48:40.002545 17928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:37223
I20260619 04:48:40.008340 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:48:40.025992 18253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.013193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.3.243.190:42719)
0619 04:48:40.013424 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.013431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.013834 (+ 403us) server_negotiation.cc:407] Connection header received
0619 04:48:40.014019 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.014025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.014104 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.014204 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:40.015586 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.016387 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.017713 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.018146 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.018280 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.020660 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:40.020666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:40.020671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:40.020711 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:40.023203 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:40.023870 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:40.023877 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:40.023881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:40.024052 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:40.024522 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:40.024528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:40.024531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:40.024847 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:40.025106 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.025429 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.025637 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260619 04:48:40.029842 17928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:48:40.036639 17928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:48:40.062011 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.048966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.3.243.131:40083)
0619 04:48:40.053091 (+ 4125us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.053098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.053131 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:48:40.053243 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.053252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.053524 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.054466 (+ 942us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:40.055194 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.057183 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.059611 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.060956 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.061092 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.061329 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.061562 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.061709 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3962,"thread_start_us":3818,"threads_started":1}
I20260619 04:48:40.062141 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.047908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:40083 (local address 127.0.0.1:38394)
0619 04:48:40.048852 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:40.048896 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:40.049112 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:40.054466 (+ 5354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:40.054473 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:40.054498 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:40.054957 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.054980 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.057337 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.057342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.059451 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.059464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.061137 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.061143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.061319 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.061379 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 04:48:40.061518 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 04:48:40.061691 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":6954,"threads_started":1}
I20260619 04:48:40.063355 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.047914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:38681 (local address 127.0.0.1:39502)
0619 04:48:40.054182 (+ 6268us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:40.054202 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:40.054316 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:40.055174 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:40.055180 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:40.055209 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:40.055577 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.055589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.057391 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.057396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.059957 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.059972 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.062509 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.062518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.062704 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.062738 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:48:40.062832 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 04:48:40.062947 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6117,"mutex_wait_us":148,"thread_start_us":373,"threads_started":1}
I20260619 04:48:40.063370 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.047945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39502 (local address 127.3.243.130:38681)
0619 04:48:40.054520 (+ 6575us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.054532 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.054566 (+ 34us) server_negotiation.cc:407] Connection header received
0619 04:48:40.054676 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.054687 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.054937 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.055127 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:40.055799 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.057267 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.060168 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.061570 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.061694 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.061871 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.062893 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.063056 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6390,"thread_start_us":6233,"threads_started":1}
I20260619 04:48:40.063557 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.048610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55220)
0619 04:48:40.055175 (+ 6565us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:40.055190 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:40.055310 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:40.056200 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:40.056205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:40.056228 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:40.056673 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.056688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.058447 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.058453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.060675 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.060689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.062710 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.062715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.062902 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.062929 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 04:48:40.062990 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 04:48:40.063102 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6450,"mutex_wait_us":15,"thread_start_us":5240,"threads_started":1}
I20260619 04:48:40.064311 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.048705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.3.243.129:33203)
0619 04:48:40.052639 (+ 3934us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.052650 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.055377 (+ 2727us) server_negotiation.cc:407] Connection header received
0619 04:48:40.055485 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.055493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.055726 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.055931 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:40.056861 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.058273 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.060856 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.062256 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.062388 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.062586 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.063797 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.063981 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3754,"thread_start_us":4557,"threads_started":1}
I20260619 04:48:40.066533 18320 tablet_service.cc:1467] Processing CreateTablet for tablet b92c810d552a4fdf880f43544b20759b (DEFAULT_TABLE table=test-table [id=caaabe05780d470db82779765db4f90c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:48:40.067194 18186 tablet_service.cc:1467] Processing CreateTablet for tablet b92c810d552a4fdf880f43544b20759b (DEFAULT_TABLE table=test-table [id=caaabe05780d470db82779765db4f90c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:48:40.068058 18320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92c810d552a4fdf880f43544b20759b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:40.068526 18186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92c810d552a4fdf880f43544b20759b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:40.068286 18052 tablet_service.cc:1467] Processing CreateTablet for tablet b92c810d552a4fdf880f43544b20759b (DEFAULT_TABLE table=test-table [id=caaabe05780d470db82779765db4f90c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:48:40.069343 18052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92c810d552a4fdf880f43544b20759b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:40.080070 18403 tablet_bootstrap.cc:492] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: Bootstrap starting.
I20260619 04:48:40.082530 18404 tablet_bootstrap.cc:492] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: Bootstrap starting.
I20260619 04:48:40.084254 18405 tablet_bootstrap.cc:492] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: Bootstrap starting.
I20260619 04:48:40.084751 18403 tablet_bootstrap.cc:654] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:40.086066 18403 log.cc:824] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:40.086524 18404 tablet_bootstrap.cc:654] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:40.087141 18405 tablet_bootstrap.cc:654] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:40.087738 18404 log.cc:824] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:40.088397 18405 log.cc:824] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:40.089906 18403 tablet_bootstrap.cc:492] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: No bootstrap required, opened a new log
I20260619 04:48:40.090189 18403 ts_tablet_manager.cc:1397] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 04:48:40.090706 18404 tablet_bootstrap.cc:492] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: No bootstrap required, opened a new log
I20260619 04:48:40.090952 18404 ts_tablet_manager.cc:1397] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 04:48:40.098243 18405 tablet_bootstrap.cc:492] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: No bootstrap required, opened a new log
I20260619 04:48:40.098517 18405 ts_tablet_manager.cc:1397] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.007s
I20260619 04:48:40.101310 18403 raft_consensus.cc:348] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.101310 18404 raft_consensus.cc:348] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.101758 18403 raft_consensus.cc:374] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:40.101751 18404 raft_consensus.cc:374] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:40.101850 18403 raft_consensus.cc:729] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8545b9f0426a4b84bd02876f772754d3, State: Initialized, Role: FOLLOWER
I20260619 04:48:40.101850 18404 raft_consensus.cc:729] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01205c6127cc44bcbbffe4168838e233, State: Initialized, Role: FOLLOWER
I20260619 04:48:40.102481 18404 consensus_queue.cc:260] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.102622 18403 consensus_queue.cc:260] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.103699 18385 heartbeater.cc:503] Master 127.3.243.190:42719 was elected leader, sending a full tablet report...
I20260619 04:48:40.104723 18404 ts_tablet_manager.cc:1428] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: Time spent starting tablet: real 0.014s user 0.005s sys 0.010s
I20260619 04:48:40.106248 18403 ts_tablet_manager.cc:1428] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260619 04:48:40.107342 18405 raft_consensus.cc:348] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.107777 18405 raft_consensus.cc:374] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:40.107877 18405 raft_consensus.cc:729] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1658cbafb44190845c63f6d446090c, State: Initialized, Role: FOLLOWER
I20260619 04:48:40.108505 18405 consensus_queue.cc:260] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.109623 18251 heartbeater.cc:503] Master 127.3.243.190:42719 was elected leader, sending a full tablet report...
I20260619 04:48:40.110208 18405 ts_tablet_manager.cc:1428] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s
W20260619 04:48:40.209641 18118 tablet.cc:2369] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:48:40.227612 18386 tablet.cc:2369] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:48:40.229641 18252 tablet.cc:2369] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:48:40.470669 18410 raft_consensus.cc:482] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:40.470954 18410 raft_consensus.cc:504] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.472929 18410 leader_election.cc:288] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681), 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
I20260619 04:48:40.474057 18409 raft_consensus.cc:482] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:40.474345 18409 raft_consensus.cc:504] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.484015 18409 leader_election.cc:288] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681), 8545b9f0426a4b84bd02876f772754d3 (127.3.243.129:33203)
I20260619 04:48:40.484304 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.473680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:45441 (local address 127.3.243.131:40083)
0619 04:48:40.473943 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.473952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.475274 (+ 1322us) server_negotiation.cc:407] Connection header received
0619 04:48:40.475585 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.475590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.475659 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.475719 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:40.476549 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.477693 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.480887 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.482699 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.482871 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.482983 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.483721 (+ 738us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.483939 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 04:48:40.485936 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.473560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:40083 (local address 127.3.243.129:45441)
0619 04:48:40.475123 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:40.475141 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:40.475403 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:40.475910 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:40.475915 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:40.475938 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:40.476343 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.476357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.477904 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.477912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.480669 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.480687 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.483228 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.483237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.483488 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.483541 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 04:48:40.484056 (+ 515us) client_negotiation.cc:240] Negotiation successful
0619 04:48:40.484989 (+ 933us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1439,"thread_start_us":1088,"threads_started":1}
I20260619 04:48:40.487285 18340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92c810d552a4fdf880f43544b20759b" candidate_uuid: "8545b9f0426a4b84bd02876f772754d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01205c6127cc44bcbbffe4168838e233" is_pre_election: true
I20260619 04:48:40.487851 18340 raft_consensus.cc:2454] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8545b9f0426a4b84bd02876f772754d3 in term 0.
I20260619 04:48:40.488862 18005 leader_election.cc:302] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01205c6127cc44bcbbffe4168838e233, 8545b9f0426a4b84bd02876f772754d3; no voters:
I20260619 04:48:40.489085 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.473328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:45953 (local address 127.3.243.130:38681)
0619 04:48:40.473550 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.473557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.474714 (+ 1157us) server_negotiation.cc:407] Connection header received
0619 04:48:40.475056 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.475061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.475120 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.475224 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:40.475865 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.477059 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.486139 (+ 9080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.487448 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.487566 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.487657 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.488700 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.488849 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:48:40.489373 18410 raft_consensus.cc:2781] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:48:40.489504 18410 raft_consensus.cc:482] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:48:40.489605 18410 raft_consensus.cc:3037] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:40.493417 18410 raft_consensus.cc:504] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.494696 18340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92c810d552a4fdf880f43544b20759b" candidate_uuid: "8545b9f0426a4b84bd02876f772754d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01205c6127cc44bcbbffe4168838e233"
I20260619 04:48:40.494897 18340 raft_consensus.cc:3037] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:40.495929 18410 leader_election.cc:288] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 1 election: Requested vote from peers 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681), 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
I20260619 04:48:40.496209 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.476696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:38681 (local address 127.3.243.131:34295)
0619 04:48:40.476974 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:40.476992 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:40.477088 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:40.489718 (+ 12630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:40.489723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:40.489749 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:40.489908 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.489917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.491373 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.491378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.493448 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.493461 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.495063 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.495071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.495369 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.495607 (+ 238us) client_negotiation.cc:769] Sending connection context
0619 04:48:40.495695 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 04:48:40.495839 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":186}
I20260619 04:48:40.496748 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.479077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:34295 (local address 127.3.243.130:38681)
0619 04:48:40.489318 (+ 10241us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.489324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.489339 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:48:40.489388 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.489393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.489455 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.489532 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:40.490066 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.491202 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.493575 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.494885 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.495473 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.495541 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.496199 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.496482 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10129,"thread_start_us":14344,"threads_started":1}
I20260619 04:48:40.497709 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.473506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:38681 (local address 127.3.243.129:45953)
0619 04:48:40.474587 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:40.474612 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:40.474843 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:40.475242 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:40.475246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:40.475272 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:40.475680 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.475694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.477227 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.477231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.485385 (+ 8154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.485399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.487877 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.487883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.488055 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.488086 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:48:40.496548 (+ 8462us) client_negotiation.cc:240] Negotiation successful
0619 04:48:40.496695 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"thread_start_us":634,"threads_started":1}
I20260619 04:48:40.497745 18206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92c810d552a4fdf880f43544b20759b" candidate_uuid: "01205c6127cc44bcbbffe4168838e233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f1658cbafb44190845c63f6d446090c" is_pre_election: true
I20260619 04:48:40.498337 18206 raft_consensus.cc:2454] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01205c6127cc44bcbbffe4168838e233 in term 0.
I20260619 04:48:40.498423 18205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92c810d552a4fdf880f43544b20759b" candidate_uuid: "8545b9f0426a4b84bd02876f772754d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f1658cbafb44190845c63f6d446090c" is_pre_election: true
I20260619 04:48:40.498665 18340 raft_consensus.cc:2454] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8545b9f0426a4b84bd02876f772754d3 in term 1.
I20260619 04:48:40.498737 18205 raft_consensus.cc:2454] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8545b9f0426a4b84bd02876f772754d3 in term 0.
I20260619 04:48:40.498914 18204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92c810d552a4fdf880f43544b20759b" candidate_uuid: "8545b9f0426a4b84bd02876f772754d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f1658cbafb44190845c63f6d446090c"
I20260619 04:48:40.499070 18005 leader_election.cc:302] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01205c6127cc44bcbbffe4168838e233, 8545b9f0426a4b84bd02876f772754d3; no voters:
I20260619 04:48:40.499145 18204 raft_consensus.cc:3037] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:40.499442 18410 raft_consensus.cc:2781] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:48:40.500048 18276 leader_election.cc:302] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01205c6127cc44bcbbffe4168838e233, 9f1658cbafb44190845c63f6d446090c; no voters:
I20260619 04:48:40.500209 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.478099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:42933 (local address 127.3.243.129:33203)
0619 04:48:40.478295 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.478305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.492655 (+ 14350us) server_negotiation.cc:407] Connection header received
0619 04:48:40.492885 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.492891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.492959 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.493015 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:40.494025 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.495023 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.497633 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.498912 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.499057 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.499132 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.499792 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.499953 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:48:40.500512 18409 raft_consensus.cc:2741] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 04:48:40.500720 18410 raft_consensus.cc:686] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 1 LEADER]: Becoming Leader. State: Replica: 8545b9f0426a4b84bd02876f772754d3, State: Running, Role: LEADER
I20260619 04:48:40.501084 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.477693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.3.243.131:42933)
0619 04:48:40.492536 (+ 14843us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:40.492555 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:40.492739 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:40.493169 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:40.493174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:40.493197 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:40.493844 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.493857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.495226 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.495234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.497457 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:40.497472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.499328 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.499337 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:40.499587 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.499637 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 04:48:40.500471 (+ 834us) client_negotiation.cc:240] Negotiation successful
0619 04:48:40.500654 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14720,"mutex_wait_us":198,"thread_start_us":14509,"threads_started":1}
I20260619 04:48:40.501252 18410 consensus_queue.cc:237] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } }
I20260619 04:48:40.502147 18072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b92c810d552a4fdf880f43544b20759b" candidate_uuid: "01205c6127cc44bcbbffe4168838e233" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8545b9f0426a4b84bd02876f772754d3" is_pre_election: true
I20260619 04:48:40.503165 18204 raft_consensus.cc:2454] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8545b9f0426a4b84bd02876f772754d3 in term 1.
I20260619 04:48:40.506263 17927 catalog_manager.cc:5495] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8545b9f0426a4b84bd02876f772754d3 (127.3.243.129). New cstate: current_term: 1 leader_uuid: "8545b9f0426a4b84bd02876f772754d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } health_report { overall_health: HEALTHY } } }
I20260619 04:48:40.624364 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.618479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55222 (local address 127.3.243.129:33203)
0619 04:48:40.618655 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.618659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.619102 (+ 443us) server_negotiation.cc:407] Connection header received
0619 04:48:40.619276 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.619280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.619359 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.619484 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:48:40.619891 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.620670 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.622075 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.622457 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.622546 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.623166 (+ 620us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:48:40.623684 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:48:40.623792 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.623982 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.624073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:48:40.632373 18340 raft_consensus.cc:1261] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 1 FOLLOWER]: Refusing update from remote peer 8545b9f0426a4b84bd02876f772754d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:48:40.632372 18204 raft_consensus.cc:1261] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 1 FOLLOWER]: Refusing update from remote peer 8545b9f0426a4b84bd02876f772754d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:48:40.633345 18410 consensus_queue.cc:1035] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:48:40.633685 18416 consensus_queue.cc:1035] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:48:40.644300 18420 mvcc.cc:205] Tried to move back new op lower bound from 7298435156500000768 to 7298435155978702848. Current Snapshot: MvccSnapshot[applied={T|T < 7298435156500000768}]
I20260619 04:48:40.644979 18422 mvcc.cc:205] Tried to move back new op lower bound from 7298435156500000768 to 7298435155978702848. Current Snapshot: MvccSnapshot[applied={T|T < 7298435156500000768}]
I20260619 04:48:40.657650 17927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:48:40.657925 17927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:48:40.661461 17927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Sending DeleteTablet for 3 replicas of tablet b92c810d552a4fdf880f43544b20759b
I20260619 04:48:40.662496 18320 tablet_service.cc:1514] Processing DeleteTablet for tablet b92c810d552a4fdf880f43544b20759b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:48:40 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:38394
I20260619 04:48:40.662540 18186 tablet_service.cc:1514] Processing DeleteTablet for tablet b92c810d552a4fdf880f43544b20759b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:48:40 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:39502
I20260619 04:48:40.662777 18052 tablet_service.cc:1514] Processing DeleteTablet for tablet b92c810d552a4fdf880f43544b20759b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:48:40 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:55220
I20260619 04:48:40.664500 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 17989
I20260619 04:48:40.674881 18430 tablet_replica.cc:330] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: stopping tablet replica
I20260619 04:48:40.675359 18432 tablet_replica.cc:330] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: stopping tablet replica
I20260619 04:48:40.675370 18430 raft_consensus.cc:2229] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:48:40.675735 18432 raft_consensus.cc:2229] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:48:40.675889 18430 raft_consensus.cc:2258] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:48:40.676777 18432 raft_consensus.cc:2258] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:48:40.676836 18431 tablet_replica.cc:330] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: stopping tablet replica
I20260619 04:48:40.677211 18431 raft_consensus.cc:2229] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:48:40.677664 18431 raft_consensus.cc:2258] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:48:40.678762 18432 ts_tablet_manager.cc:1905] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:48:40.679419 18431 ts_tablet_manager.cc:1905] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:48:40.686491 18431 ts_tablet_manager.cc:1918] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:48:40.686477 18432 ts_tablet_manager.cc:1918] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:48:40.686650 18431 log.cc:1196] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/b92c810d552a4fdf880f43544b20759b
I20260619 04:48:40.686657 18432 log.cc:1196] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/b92c810d552a4fdf880f43544b20759b
I20260619 04:48:40.687194 18431 ts_tablet_manager.cc:1939] T b92c810d552a4fdf880f43544b20759b P 9f1658cbafb44190845c63f6d446090c: Deleting consensus metadata
I20260619 04:48:40.687199 18432 ts_tablet_manager.cc:1939] T b92c810d552a4fdf880f43544b20759b P 8545b9f0426a4b84bd02876f772754d3: Deleting consensus metadata
I20260619 04:48:40.688990 17915 catalog_manager.cc:4841] TS 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681): tablet b92c810d552a4fdf880f43544b20759b (table test-table [id=caaabe05780d470db82779765db4f90c]) successfully deleted
I20260619 04:48:40.689492 18430 ts_tablet_manager.cc:1905] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:48:40.690125 17914 catalog_manager.cc:4841] TS 8545b9f0426a4b84bd02876f772754d3 (127.3.243.129:33203): tablet b92c810d552a4fdf880f43544b20759b (table test-table [id=caaabe05780d470db82779765db4f90c]) successfully deleted
Jun 19 04:48:40 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:48:40.695206 18430 ts_tablet_manager.cc:1918] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:48:40.695399 18430 log.cc:1196] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/b92c810d552a4fdf880f43544b20759b
I20260619 04:48:40.695935 18430 ts_tablet_manager.cc:1939] T b92c810d552a4fdf880f43544b20759b P 01205c6127cc44bcbbffe4168838e233: Deleting consensus metadata
I20260619 04:48:40.697811 17912 catalog_manager.cc:4841] TS 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083): tablet b92c810d552a4fdf880f43544b20759b (table test-table [id=caaabe05780d470db82779765db4f90c]) successfully deleted
I20260619 04:48:40.702049 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.665672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.3.243.129:33203)
0619 04:48:40.665865 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.665871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.682150 (+ 16279us) server_negotiation.cc:407] Connection header received
0619 04:48:40.682625 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.682631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.682713 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.682786 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:40.684856 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.686047 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.688489 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.689163 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.689279 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.695393 (+ 6114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:40.695409 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:40.695435 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:40.695483 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:40.699203 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:40.700073 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:40.700079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:40.700082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:40.700276 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:40.700679 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:40.700682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:40.700684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:40.701038 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:40.701259 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.701590 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.701773 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:48:40.702804 18112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:40.801133 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 17989
I20260619 04:48:40.820520 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 18123
Jun 19 04:48:40 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:48:40.836071 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.821305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39516 (local address 127.3.243.130:38681)
0619 04:48:40.821493 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.821499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.821597 (+ 98us) server_negotiation.cc:407] Connection header received
0619 04:48:40.821810 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.821817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.821901 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.822020 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:40.823418 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.824362 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.825702 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.826105 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.826249 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.830591 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:40.830604 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:40.830620 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:40.830652 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:40.833402 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:40.834068 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:40.834075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:40.834077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:40.834193 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:40.834581 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:40.834586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:40.834589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:40.834938 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:40.835250 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.835544 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.835746 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:48:40.836758 18246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:40.926147 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 18123
I20260619 04:48:40.943027 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 18257
Jun 19 04:48:40 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:48:40.957614 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:40.943809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38398 (local address 127.3.243.131:40083)
0619 04:48:40.944001 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:40.944006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:40.944024 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:48:40.944198 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:40.944202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:40.944282 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:40.944366 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:40.945593 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.946338 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.947584 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:40.947960 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:40.948066 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:40.952541 (+ 4475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:40.952556 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:40.952572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:40.952604 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:40.955150 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:40.955712 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:40.955718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:40.955720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:40.955865 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:40.956313 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:40.956316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:40.956318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:40.956653 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:40.956873 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:40.957079 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 04:48:40.957224 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:48:40.958353 18380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:41.047778 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 18257
I20260619 04:48:41.065083 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 17896
I20260619 04:48:41.066200 17958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:41.176987 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 17896
I20260619 04:48:41.198119 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42719
--webserver_interface=127.3.243.190
--webserver_port=41973
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:42719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:41.528028 18438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:41.528426 18438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:41.528595 18438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:41.535928 18438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:48:41.536057 18438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:41.536171 18438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:41.536298 18438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:48:41.536377 18438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:48:41.546271 18438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:42719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=41973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:41.547628 18438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:41.548964 18438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:41.572058 18444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:41.572088 18447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:41.572160 18445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:41.572968 18438 server_base.cc:1034] running on GCE node
I20260619 04:48:41.573765 18438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:41.575294 18438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:41.576538 18438 hybrid_clock.cc:648] HybridClock initialized: now 1781844521576491 us; error 54 us; skew 500 ppm
I20260619 04:48:41.576956 18438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:41 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844521, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:41.581396 18438 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:48:41.583210 18438 webserver.cc:460] Webserver started at http://127.3.243.190:41973/ using document root <none> and password file <none>
I20260619 04:48:41.583961 18438 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:41.584074 18438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:41.590071 18438 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:48:41.593011 18454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:41.594367 18438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:48:41.594502 18438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "999fee5983b04f05a72050eba740fada"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "2093314be382603aa216b7162f489c69"
server_key_iv: "53eb8741a19a7b21bf7acf737e126f4f"
server_key_version: "encryptionkey@0"
I20260619 04:48:41.595496 18438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:41.614738 18438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:41.619176 18438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:41.619401 18438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:41.638722 18438 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:42719
I20260619 04:48:41.638777 18505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:42719 every 8 connection(s)
I20260619 04:48:41.640413 18438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:48:41.641462 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 18438
I20260619 04:48:41.642272 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:33203
--local_ip_for_outbound_sockets=127.3.243.129
--tserver_master_addrs=127.3.243.190:42719
--webserver_port=43021
--webserver_interface=127.3.243.129
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:48:41.647778 18506 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:48:41.652168 18506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Bootstrap starting.
I20260619 04:48:41.682147 18506 log.cc:824] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:41.697779 18506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:48:41.698457 18506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Bootstrap complete.
I20260619 04:48:41.708482 18506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:41.709445 18506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999fee5983b04f05a72050eba740fada, State: Initialized, Role: FOLLOWER
I20260619 04:48:41.710238 18506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:41.710495 18506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:48:41.710620 18506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:48:41.710813 18506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:48:41.714901 18506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:41.715454 18506 leader_election.cc:302] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999fee5983b04f05a72050eba740fada; no voters:
I20260619 04:48:41.716079 18506 leader_election.cc:288] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 04:48:41.716334 18510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:48:41.717576 18510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 2 LEADER]: Becoming Leader. State: Replica: 999fee5983b04f05a72050eba740fada, State: Running, Role: LEADER
I20260619 04:48:41.718360 18506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:48:41.718222 18510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:41.721336 18512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999fee5983b04f05a72050eba740fada. Latest consensus state: current_term: 2 leader_uuid: "999fee5983b04f05a72050eba740fada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } } }
I20260619 04:48:41.721717 18512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:41.722143 18511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "999fee5983b04f05a72050eba740fada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } } }
I20260619 04:48:41.722450 18511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:41.732370 18518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:48:41.740499 18518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:48:41.741879 18518 catalog_manager.cc:1259] Loaded cluster ID: 5372fc6c60a843b9aaadf9d2304b265d
I20260619 04:48:41.741977 18518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:48:41.745038 18518 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 04:48:41.763593 18518 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 04:48:41.773485 18518 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:48:41.775161 18518 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Loaded TSK: 0
I20260619 04:48:41.776675 18518 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 04:48:42.022498 18508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:42.022837 18508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:42.023010 18508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:42.030586 18508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:42.030758 18508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:42.030941 18508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:48:42.041239 18508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:33203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=43021
--tserver_master_addrs=127.3.243.190:42719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:42.042654 18508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:42.044072 18508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:42.058791 18534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:42.059291 18536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:42.059373 18533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:42.059499 18508 server_base.cc:1034] running on GCE node
I20260619 04:48:42.060446 18508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:42.061612 18508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:42.062894 18508 hybrid_clock.cc:648] HybridClock initialized: now 1781844522062823 us; error 51 us; skew 500 ppm
I20260619 04:48:42.063273 18508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:42 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:42.067173 18508 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:48:42.068828 18508 webserver.cc:460] Webserver started at http://127.3.243.129:43021/ using document root <none> and password file <none>
I20260619 04:48:42.069703 18508 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:42.069859 18508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:42.076778 18508 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 04:48:42.080371 18543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:42.081990 18508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:48:42.082165 18508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "8545b9f0426a4b84bd02876f772754d3"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "55cfbb922111ff19061733c7cc481457"
server_key_iv: "a0d529ce50316f1dbeeba8447af40a56"
server_key_version: "encryptionkey@0"
I20260619 04:48:42.083463 18508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:42.112078 18508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:42.116053 18508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:42.116339 18508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:42.117301 18508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:42.118656 18508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:42.118748 18508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:42.118824 18508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:42.118880 18508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:42.152019 18508 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:33203
I20260619 04:48:42.152112 18655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:33203 every 8 connection(s)
I20260619 04:48:42.153519 18508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:48:42.160980 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 18508
I20260619 04:48:42.161818 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:38681
--local_ip_for_outbound_sockets=127.3.243.130
--tserver_master_addrs=127.3.243.190:42719
--webserver_port=45059
--webserver_interface=127.3.243.130
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:48:42 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:42.179467 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:42.156758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:39117)
0619 04:48:42.157584 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:42.157629 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:42.159372 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:42.160977 (+ 1605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:42.160987 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:42.161766 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:42.162618 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:42.162647 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:42.164298 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:42.164306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:42.166522 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:42.166541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:42.167560 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:42.167569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:42.167886 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:42.168643 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:42.168699 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:42.171046 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:42.175571 (+ 4525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:42.175578 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:42.175589 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:42.175987 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:42.176480 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:42.176483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:42.176485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:42.176671 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:42.177581 (+ 910us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:42.177587 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:42.177877 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 04:48:42.178102 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 04:48:42.178525 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":355,"threads_started":1}
I20260619 04:48:42.179976 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:42.156516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:39117 (local address 127.3.243.190:42719)
0619 04:48:42.157130 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:42.157142 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:42.157753 (+ 611us) server_negotiation.cc:407] Connection header received
0619 04:48:42.159998 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:42.160011 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:42.160438 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:42.160733 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:42.162866 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:42.164085 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:42.166675 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:42.167319 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:42.167444 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:42.171321 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:42.171334 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:42.171355 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:42.171411 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:42.175434 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:42.176141 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:42.176152 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:42.176162 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:42.176360 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:42.176786 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:42.176801 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:42.176807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:42.177435 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:42.177629 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:42.178372 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 04:48:42.178806 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
I20260619 04:48:42.183416 18656 heartbeater.cc:347] Connected to a master server at 127.3.243.190:42719
I20260619 04:48:42.183699 18656 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:42.184702 18656 heartbeater.cc:511] Master 127.3.243.190:42719 requested a full tablet report, sending...
I20260619 04:48:42.187556 18471 ts_manager.cc:194] Registered new tserver with Master: 8545b9f0426a4b84bd02876f772754d3 (127.3.243.129:33203)
I20260619 04:48:42.189901 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:39117
W20260619 04:48:42.496639 18660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:42.496964 18660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:42.497134 18660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:42.504609 18660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:42.504755 18660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:42.504933 18660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:48:42.514838 18660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=45059
--tserver_master_addrs=127.3.243.190:42719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:42.516172 18660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:42.517520 18660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:42.531265 18666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:42.531209 18665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:42.531771 18668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:42.532069 18660 server_base.cc:1034] running on GCE node
I20260619 04:48:42.532776 18660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:42.533784 18660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:42.535013 18660 hybrid_clock.cc:648] HybridClock initialized: now 1781844522534959 us; error 64 us; skew 500 ppm
I20260619 04:48:42.535329 18660 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:42 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:42.538393 18660 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:48:42.539858 18660 webserver.cc:460] Webserver started at http://127.3.243.130:45059/ using document root <none> and password file <none>
I20260619 04:48:42.540607 18660 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:42.540777 18660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:42.545986 18660 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:48:42.548718 18675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:42.550119 18660 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:48:42.550293 18660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "9f1658cbafb44190845c63f6d446090c"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "431c5d0db143535e0e809daf19845b91"
server_key_iv: "256114625a1110da429b025be321fb20"
server_key_version: "encryptionkey@0"
I20260619 04:48:42.551247 18660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:42.563859 18660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:42.567556 18660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:42.567842 18660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:42.568825 18660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:42.570026 18660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:42.570163 18660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:42.570286 18660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:42.570366 18660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:42.604979 18660 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:38681
I20260619 04:48:42.605033 18787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:38681 every 8 connection(s)
I20260619 04:48:42.606487 18660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:48:42.609637 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 18660
I20260619 04:48:42.610224 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:40083
--local_ip_for_outbound_sockets=127.3.243.131
--tserver_master_addrs=127.3.243.190:42719
--webserver_port=36537
--webserver_interface=127.3.243.131
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:48:42 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:42.627741 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:42.609095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:35767 (local address 127.3.243.190:42719)
0619 04:48:42.609313 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:42.609320 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:42.610718 (+ 1398us) server_negotiation.cc:407] Connection header received
0619 04:48:42.612711 (+ 1993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:42.612716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:42.612803 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:42.612925 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:42.614374 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:42.615059 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:42.616682 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:42.617122 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:42.617271 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:42.621863 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:42.621868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:42.621871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:42.621905 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:42.624532 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:42.625439 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:42.625443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:42.625445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:42.625564 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:42.626048 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:42.626051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:42.626053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:42.626305 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:42.626448 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:42.627332 (+ 884us) server_negotiation.cc:299] Negotiation successful
0619 04:48:42.627507 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:48:42.629406 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:42.609637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:35767)
0619 04:48:42.610553 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:42.610603 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:42.612427 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:42.613179 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:42.613188 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:42.613663 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:42.614202 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:42.614220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:42.615216 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:42.615223 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:42.616490 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:42.616502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:42.617354 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:42.617361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:42.617664 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:42.618424 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:42.618460 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:42.621612 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:42.624674 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:42.624686 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:42.624714 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:42.625321 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:42.625679 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:42.625687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:42.625701 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:42.625938 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:42.626460 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:42.626471 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:42.626930 (+ 459us) client_negotiation.cc:769] Sending connection context
0619 04:48:42.627238 (+ 308us) client_negotiation.cc:240] Negotiation successful
0619 04:48:42.627811 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":349,"threads_started":1}
I20260619 04:48:42.631119 18788 heartbeater.cc:347] Connected to a master server at 127.3.243.190:42719
I20260619 04:48:42.631425 18788 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:42.632186 18788 heartbeater.cc:511] Master 127.3.243.190:42719 requested a full tablet report, sending...
I20260619 04:48:42.633755 18471 ts_manager.cc:194] Registered new tserver with Master: 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681)
I20260619 04:48:42.634536 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:35767
W20260619 04:48:42.940775 18791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:42.941092 18791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:42.941273 18791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:42.948446 18791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:42.948585 18791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:42.948765 18791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:48:42.958467 18791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:40083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=36537
--tserver_master_addrs=127.3.243.190:42719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:42.959724 18791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:42.961053 18791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:42.974203 18799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:42.974255 18796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:42.974278 18797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:42.975461 18791 server_base.cc:1034] running on GCE node
I20260619 04:48:42.976022 18791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:42.976979 18791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:42.978240 18791 hybrid_clock.cc:648] HybridClock initialized: now 1781844522978197 us; error 59 us; skew 500 ppm
I20260619 04:48:42.978538 18791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:42 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:42.981518 18791 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:48:42.982878 18791 webserver.cc:460] Webserver started at http://127.3.243.131:36537/ using document root <none> and password file <none>
I20260619 04:48:42.983523 18791 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:42.983671 18791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:42.989089 18791 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:48:42.991758 18806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:42.993168 18791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:48:42.993387 18791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "01205c6127cc44bcbbffe4168838e233"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "a6db3955180c1edd91392a0870d277ee"
server_key_iv: "4fa463a5998b0ff67862c3558f9bff6e"
server_key_version: "encryptionkey@0"
I20260619 04:48:42.994709 18791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:43.019155 18791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:43.023221 18791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:43.023523 18791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:43.024545 18791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:43.025892 18791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:43.026010 18791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:43.026093 18791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:43.026149 18791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:43.059278 18791 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:40083
I20260619 04:48:43.059340 18918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:40083 every 8 connection(s)
I20260619 04:48:43.060866 18791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:48:43.066946 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 18791
Jun 19 04:48:43 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844522, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:48:43.085017 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.063491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:37373 (local address 127.3.243.190:42719)
0619 04:48:43.063660 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.063664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.065002 (+ 1338us) server_negotiation.cc:407] Connection header received
0619 04:48:43.066826 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.066832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.066923 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.067052 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:43.069224 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.070385 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.072677 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.073406 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.073510 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.077555 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:43.077564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:43.077570 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:43.077612 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:43.081124 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.082087 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.082091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.082093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.082280 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.082754 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.082760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.082763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.083217 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:43.083348 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.084383 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.084616 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:48:43.085448 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.067888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.3.243.190:42719)
0619 04:48:43.068382 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.068387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.069745 (+ 1358us) server_negotiation.cc:407] Connection header received
0619 04:48:43.069919 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.069924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.070031 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.070228 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:43.072160 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.073277 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.075193 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.075617 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.075721 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.078992 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:43.079000 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:43.079003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:43.079049 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:43.082364 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.083212 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.083215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.083217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.083360 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.083807 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.083814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.083820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.084188 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:43.084434 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.084820 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.085027 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":274,"threads_started":1}
I20260619 04:48:43.086078 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.063935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:37373)
0619 04:48:43.064833 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.064878 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:43.066545 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:43.067300 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:43.067308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:43.068043 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:43.069036 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.069066 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.070578 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.070585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.072531 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.072543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.073634 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.073642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.073922 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.074764 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:43.074794 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:43.077310 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:43.081309 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:43.081319 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:43.081335 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:43.081945 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:43.082396 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:43.082404 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:43.082410 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:43.082623 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:43.083353 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:43.083360 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:43.083789 (+ 429us) client_negotiation.cc:769] Sending connection context
0619 04:48:43.084157 (+ 368us) client_negotiation.cc:240] Negotiation successful
0619 04:48:43.084633 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":416,"threads_started":1}
I20260619 04:48:43.087491 18919 heartbeater.cc:347] Connected to a master server at 127.3.243.190:42719
I20260619 04:48:43.087822 18919 heartbeater.cc:464] Registering TS with master...
I20260619 04:48:43.088717 18919 heartbeater.cc:511] Master 127.3.243.190:42719 requested a full tablet report, sending...
I20260619 04:48:43.090018 18471 ts_manager.cc:194] Registered new tserver with Master: 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
I20260619 04:48:43.090843 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:37373
I20260619 04:48:43.100576 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:48:43.107441 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.101554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.3.243.190:42719)
0619 04:48:43.101780 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.101788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.102120 (+ 332us) server_negotiation.cc:407] Connection header received
0619 04:48:43.102377 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.102381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.102508 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.102663 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:48:43.103114 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.103796 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.105278 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.105687 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.105819 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.106367 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:48:43.106862 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:48:43.107025 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.107084 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.107161 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:48:43.108450 18471 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:48:43.115643 18471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:48:43.139073 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.128521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:40083 (local address 127.0.0.1:38402)
0619 04:48:43.131234 (+ 2713us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.131255 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:43.131361 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:43.131969 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:43.131977 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:43.132003 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:43.132455 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.132464 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.134122 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.134129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.136150 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.136171 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.138061 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.138069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.138264 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.138324 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 04:48:43.138432 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 04:48:43.138629 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2597,"mutex_wait_us":37,"thread_start_us":1506,"threads_started":1}
I20260619 04:48:43.139441 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.127369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.3.243.130:38681)
0619 04:48:43.129560 (+ 2191us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.129569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.130614 (+ 1045us) server_negotiation.cc:407] Connection header received
0619 04:48:43.130844 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.130855 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.131159 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.131369 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:43.132195 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.133537 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.136612 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.138028 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.138165 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.138367 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.138895 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.139090 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2024,"thread_start_us":1907,"threads_started":1}
I20260619 04:48:43.140072 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.127024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:38681 (local address 127.0.0.1:39522)
0619 04:48:43.130375 (+ 3351us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.130412 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:43.130573 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:43.131397 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:43.131409 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:43.131444 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:43.131907 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.131919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.133705 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.133711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.136453 (+ 2742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.136468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.138520 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.138525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.138675 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.138703 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:48:43.138817 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 04:48:43.138937 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3102,"thread_start_us":2935,"threads_started":1}
I20260619 04:48:43.140331 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.128077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.3.243.131:40083)
0619 04:48:43.128744 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.128753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.131487 (+ 2734us) server_negotiation.cc:407] Connection header received
0619 04:48:43.131579 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.131586 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.131744 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.132598 (+ 854us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:43.132921 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.133997 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.136370 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.137827 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.139172 (+ 1345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.139420 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.139507 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.139972 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":822,"threads_started":1}
I20260619 04:48:43.140822 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.127051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55234)
0619 04:48:43.130376 (+ 3325us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.130413 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:43.130679 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:43.131793 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:43.131799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:43.131838 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:43.134570 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.134582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.136504 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.136512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.137933 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.137945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.139868 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.139876 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.140079 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.140159 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 04:48:43.140265 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 04:48:43.140391 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3141,"mutex_wait_us":2111,"thread_start_us":3896,"threads_started":1}
I20260619 04:48:43.141641 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.127569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.3.243.129:33203)
0619 04:48:43.128201 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.128209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.130525 (+ 2316us) server_negotiation.cc:407] Connection header received
0619 04:48:43.130955 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.130964 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.131268 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.131459 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:43.134798 (+ 3339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.136324 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.138115 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.139317 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.139476 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.139701 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.141097 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.141290 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":1143,"threads_started":1}
I20260619 04:48:43.144279 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7f8acf1995490f84fe79abc57443c9 (DEFAULT_TABLE table=test-table [id=df24752bf2634b7e991de45f78d638ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:48:43.144279 18854 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7f8acf1995490f84fe79abc57443c9 (DEFAULT_TABLE table=test-table [id=df24752bf2634b7e991de45f78d638ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:48:43.145944 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7f8acf1995490f84fe79abc57443c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:43.146222 18854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7f8acf1995490f84fe79abc57443c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:43.147408 18591 tablet_service.cc:1467] Processing CreateTablet for tablet 2b7f8acf1995490f84fe79abc57443c9 (DEFAULT_TABLE table=test-table [id=df24752bf2634b7e991de45f78d638ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:48:43.148590 18591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b7f8acf1995490f84fe79abc57443c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:48:43.164880 18935 tablet_bootstrap.cc:492] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: Bootstrap starting.
I20260619 04:48:43.165794 18936 tablet_bootstrap.cc:492] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: Bootstrap starting.
I20260619 04:48:43.167088 18937 tablet_bootstrap.cc:492] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Bootstrap starting.
I20260619 04:48:43.170285 18937 tablet_bootstrap.cc:654] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:43.170503 18935 tablet_bootstrap.cc:654] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:43.171118 18936 tablet_bootstrap.cc:654] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: Neither blocks nor log segments found. Creating new log.
I20260619 04:48:43.171414 18937 log.cc:824] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:43.171837 18935 log.cc:824] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:43.172365 18936 log.cc:824] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:43.174571 18937 tablet_bootstrap.cc:492] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: No bootstrap required, opened a new log
I20260619 04:48:43.174616 18935 tablet_bootstrap.cc:492] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: No bootstrap required, opened a new log
I20260619 04:48:43.174881 18935 ts_tablet_manager.cc:1397] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 04:48:43.174912 18937 ts_tablet_manager.cc:1397] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 04:48:43.175595 18936 tablet_bootstrap.cc:492] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: No bootstrap required, opened a new log
I20260619 04:48:43.175851 18936 ts_tablet_manager.cc:1397] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260619 04:48:43.182324 18935 raft_consensus.cc:348] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:43.182672 18935 raft_consensus.cc:374] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:43.182798 18935 raft_consensus.cc:729] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1658cbafb44190845c63f6d446090c, State: Initialized, Role: FOLLOWER
I20260619 04:48:43.183429 18935 consensus_queue.cc:260] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:43.184604 18788 heartbeater.cc:503] Master 127.3.243.190:42719 was elected leader, sending a full tablet report...
I20260619 04:48:43.185083 18935 ts_tablet_manager.cc:1428] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260619 04:48:43.185900 18937 raft_consensus.cc:348] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:43.186406 18937 raft_consensus.cc:374] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:43.186513 18937 raft_consensus.cc:729] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8545b9f0426a4b84bd02876f772754d3, State: Initialized, Role: FOLLOWER
I20260619 04:48:43.187165 18937 consensus_queue.cc:260] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:43.188355 18656 heartbeater.cc:503] Master 127.3.243.190:42719 was elected leader, sending a full tablet report...
I20260619 04:48:43.189049 18937 ts_tablet_manager.cc:1428] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260619 04:48:43.189867 18936 raft_consensus.cc:348] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:43.190358 18936 raft_consensus.cc:374] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:48:43.190474 18936 raft_consensus.cc:729] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01205c6127cc44bcbbffe4168838e233, State: Initialized, Role: FOLLOWER
I20260619 04:48:43.191085 18936 consensus_queue.cc:260] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:43.192693 18919 heartbeater.cc:503] Master 127.3.243.190:42719 was elected leader, sending a full tablet report...
I20260619 04:48:43.193205 18936 ts_tablet_manager.cc:1428] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260619 04:48:43.193683 18941 raft_consensus.cc:482] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:43.193964 18941 raft_consensus.cc:504] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:43.195762 18941 leader_election.cc:288] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8545b9f0426a4b84bd02876f772754d3 (127.3.243.129:33203), 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
I20260619 04:48:43.203853 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.196404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:41895 (local address 127.3.243.129:33203)
0619 04:48:43.196631 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.196639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.197279 (+ 640us) server_negotiation.cc:407] Connection header received
0619 04:48:43.197515 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.197524 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.197616 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.197736 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:43.198677 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.199729 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.201724 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.202971 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.203094 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.203150 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.203547 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.203664 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 04:48:43.204051 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.196178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.3.243.130:41895)
0619 04:48:43.197171 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.197188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:43.197358 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:43.197780 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:43.197786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:43.197816 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:43.198241 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.198254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.199865 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.199871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.201560 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.201571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.203171 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.203178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.203344 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.203381 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 04:48:43.203479 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 04:48:43.203638 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":3,"thread_start_us":667,"threads_started":1}
I20260619 04:48:43.204967 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.196424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:50417 (local address 127.3.243.131:40083)
0619 04:48:43.196616 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.196624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.198095 (+ 1471us) server_negotiation.cc:407] Connection header received
0619 04:48:43.198245 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.198250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.198325 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.198410 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:48:43.199463 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.200542 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.202812 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.203986 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.204176 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.204252 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.204688 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.204807 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:48:43.205171 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.196192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:40083 (local address 127.3.243.130:50417)
0619 04:48:43.197993 (+ 1801us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.198008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:43.198121 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:43.198795 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:43.198800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:43.198827 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:48:43.199307 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.199321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.200687 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.200691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.202634 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:43.202651 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.204384 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.204392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:43.204543 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.204565 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 04:48:43.204636 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 04:48:43.204764 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1635,"mutex_wait_us":48,"spinlock_wait_cycles":15616,"thread_start_us":1432,"threads_started":1}
I20260619 04:48:43.205202 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7f8acf1995490f84fe79abc57443c9" candidate_uuid: "9f1658cbafb44190845c63f6d446090c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8545b9f0426a4b84bd02876f772754d3" is_pre_election: true
I20260619 04:48:43.205753 18611 raft_consensus.cc:2454] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f1658cbafb44190845c63f6d446090c in term 0.
I20260619 04:48:43.206066 18874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7f8acf1995490f84fe79abc57443c9" candidate_uuid: "9f1658cbafb44190845c63f6d446090c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01205c6127cc44bcbbffe4168838e233" is_pre_election: true
I20260619 04:48:43.206447 18678 leader_election.cc:302] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8545b9f0426a4b84bd02876f772754d3, 9f1658cbafb44190845c63f6d446090c; no voters:
I20260619 04:48:43.206516 18874 raft_consensus.cc:2454] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f1658cbafb44190845c63f6d446090c in term 0.
I20260619 04:48:43.206848 18941 raft_consensus.cc:2781] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:48:43.207011 18941 raft_consensus.cc:482] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:48:43.207140 18941 raft_consensus.cc:3037] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:43.209434 18941 raft_consensus.cc:504] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:43.210093 18941 leader_election.cc:288] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [CANDIDATE]: Term 1 election: Requested vote from peers 8545b9f0426a4b84bd02876f772754d3 (127.3.243.129:33203), 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
I20260619 04:48:43.210387 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7f8acf1995490f84fe79abc57443c9" candidate_uuid: "9f1658cbafb44190845c63f6d446090c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8545b9f0426a4b84bd02876f772754d3"
I20260619 04:48:43.210594 18611 raft_consensus.cc:3037] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:43.210572 18874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b7f8acf1995490f84fe79abc57443c9" candidate_uuid: "9f1658cbafb44190845c63f6d446090c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01205c6127cc44bcbbffe4168838e233"
I20260619 04:48:43.210758 18874 raft_consensus.cc:3037] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:48:43.212973 18611 raft_consensus.cc:2454] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1658cbafb44190845c63f6d446090c in term 1.
I20260619 04:48:43.213028 18874 raft_consensus.cc:2454] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f1658cbafb44190845c63f6d446090c in term 1.
I20260619 04:48:43.213358 18678 leader_election.cc:302] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8545b9f0426a4b84bd02876f772754d3, 9f1658cbafb44190845c63f6d446090c; no voters:
I20260619 04:48:43.213606 18941 raft_consensus.cc:2781] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:48:43.214007 18941 raft_consensus.cc:686] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 1 LEADER]: Becoming Leader. State: Replica: 9f1658cbafb44190845c63f6d446090c, State: Running, Role: LEADER
I20260619 04:48:43.214447 18941 consensus_queue.cc:237] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:43.218750 18471 catalog_manager.cc:5495] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f1658cbafb44190845c63f6d446090c (127.3.243.130). New cstate: current_term: 1 leader_uuid: "9f1658cbafb44190845c63f6d446090c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } health_report { overall_health: UNKNOWN } } }
I20260619 04:48:43.238513 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.232978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.3.243.130:38681)
0619 04:48:43.233299 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.233306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.233422 (+ 116us) server_negotiation.cc:407] Connection header received
0619 04:48:43.233656 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.233660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.233739 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.233891 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:48:43.234255 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.234936 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.236433 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.236842 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.236981 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.237478 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:48:43.237949 (+ 471us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:48:43.238106 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.238161 (+ 55us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.238239 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260619 04:48:43.246563 18874 raft_consensus.cc:1261] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [term 1 FOLLOWER]: Refusing update from remote peer 9f1658cbafb44190845c63f6d446090c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:48:43.246568 18611 raft_consensus.cc:1261] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Refusing update from remote peer 9f1658cbafb44190845c63f6d446090c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:48:43.247614 18946 consensus_queue.cc:1035] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:48:43.247974 18941 consensus_queue.cc:1035] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [LEADER]: Connected to new peer: Peer: permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:48:43.258073 18949 mvcc.cc:205] Tried to move back new op lower bound from 7298435167207325696 to 7298435167090110464. Current Snapshot: MvccSnapshot[applied={T|T < 7298435167207325696}]
I20260619 04:48:43.261251 18950 mvcc.cc:205] Tried to move back new op lower bound from 7298435167207325696 to 7298435167090110464. Current Snapshot: MvccSnapshot[applied={T|T < 7298435167207325696}]
I20260619 04:48:43.272337 18471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:48:43.272596 18471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:48:43.275722 18471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Sending DeleteTablet for 3 replicas of tablet 2b7f8acf1995490f84fe79abc57443c9
I20260619 04:48:43.276809 18723 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b7f8acf1995490f84fe79abc57443c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:48:43 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:39522
I20260619 04:48:43.276903 18591 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b7f8acf1995490f84fe79abc57443c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:48:43 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:55234
I20260619 04:48:43.277122 18854 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b7f8acf1995490f84fe79abc57443c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:48:43 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:38402
I20260619 04:48:43.278015 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 18508
I20260619 04:48:43.297966 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.278897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.3.243.129:33203)
0619 04:48:43.279044 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.279050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.279187 (+ 137us) server_negotiation.cc:407] Connection header received
0619 04:48:43.279378 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.279387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.279515 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.279615 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:43.281315 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.282849 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.284779 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.285742 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.285873 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.289492 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:43.289514 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:43.289539 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:43.289586 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:43.294515 (+ 4929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.295393 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.295406 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.295412 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.295599 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.296042 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.296050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.296056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.296708 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:43.297012 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.297340 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.297557 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 04:48:43.298591 18959 tablet_replica.cc:330] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: stopping tablet replica
I20260619 04:48:43.298960 18960 tablet_replica.cc:330] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: stopping tablet replica
I20260619 04:48:43.298995 18651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:43.299521 18960 raft_consensus.cc:2229] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:48:43.299805 18958 tablet_replica.cc:330] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: stopping tablet replica
I20260619 04:48:43.300143 18960 raft_consensus.cc:2258] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:48:43.300287 18958 raft_consensus.cc:2229] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:48:43.301074 18958 raft_consensus.cc:2258] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:48:43.302747 18960 ts_tablet_manager.cc:1905] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:48:43.303082 18958 ts_tablet_manager.cc:1905] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:48:43.309775 18960 ts_tablet_manager.cc:1918] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:48:43.310274 18958 ts_tablet_manager.cc:1918] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:48:43.310575 18958 log.cc:1196] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/2b7f8acf1995490f84fe79abc57443c9
I20260619 04:48:43.311204 18960 log.cc:1196] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/2b7f8acf1995490f84fe79abc57443c9
I20260619 04:48:43.311417 18958 ts_tablet_manager.cc:1939] T 2b7f8acf1995490f84fe79abc57443c9 P 9f1658cbafb44190845c63f6d446090c: Deleting consensus metadata
I20260619 04:48:43.311775 18960 ts_tablet_manager.cc:1939] T 2b7f8acf1995490f84fe79abc57443c9 P 01205c6127cc44bcbbffe4168838e233: Deleting consensus metadata
I20260619 04:48:43.313649 18458 catalog_manager.cc:4841] TS 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681): tablet 2b7f8acf1995490f84fe79abc57443c9 (table test-table [id=df24752bf2634b7e991de45f78d638ff]) successfully deleted
I20260619 04:48:43.314618 18455 catalog_manager.cc:4841] TS 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083): tablet 2b7f8acf1995490f84fe79abc57443c9 (table test-table [id=df24752bf2634b7e991de45f78d638ff]) successfully deleted
I20260619 04:48:43.299180 18959 raft_consensus.cc:2229] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:48:43.400866 18959 raft_consensus.cc:2258] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:48:43.400938 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 18508
W20260619 04:48:43.424849 18457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.129:33203 (error 108)
I20260619 04:48:43.425077 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 18660
I20260619 04:48:43.427394 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.426989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55252)
0619 04:48:43.427156 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.427263 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 04:48:43.427696 18457 catalog_manager.cc:4584] TS 8545b9f0426a4b84bd02876f772754d3 (127.3.243.129:33203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b7f8acf1995490f84fe79abc57443c9: Network error: Client connection negotiation failed: client connection to 127.3.243.129:33203: connect: Connection refused (error 111)
I20260619 04:48:43.445699 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.425958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.3.243.130:38681)
0619 04:48:43.426157 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.426165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.426455 (+ 290us) server_negotiation.cc:407] Connection header received
0619 04:48:43.426541 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.426545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.426633 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.426730 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:43.428466 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.429265 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.431242 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.431700 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.431807 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.439662 (+ 7855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:43.439676 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:43.439691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:43.439734 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:43.442820 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.443512 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.443527 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.443530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.443677 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.444070 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.444079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.444082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.444511 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:43.444850 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.445111 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.445332 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:48:43.446401 18783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:43.481223 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.480769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55268)
0619 04:48:43.480974 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.481068 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 04:48:43.482331 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.481990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55282)
0619 04:48:43.482136 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.482224 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:48:43.542922 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.542417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55298)
0619 04:48:43.542663 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.542755 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260619 04:48:43.543926 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.543572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55302)
0619 04:48:43.543733 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.543812 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 04:48:43.558516 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 18660
I20260619 04:48:43.577247 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 18791
I20260619 04:48:43.591409 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.578073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.3.243.131:40083)
0619 04:48:43.578244 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:43.578249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:43.578266 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:48:43.578384 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:43.578388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:43.578452 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:43.578534 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:43.579784 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.580583 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.581844 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:43.582230 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:43.582318 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:43.584996 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:43.585009 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:43.585025 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:43.585062 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:43.589018 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.589651 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.589658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.589660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.589783 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:43.590099 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:43.590102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:43.590104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:43.590428 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:43.590636 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:43.590905 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 04:48:43.591055 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:48:43.592025 18914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:43.624513 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.623904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55316)
0619 04:48:43.624184 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.624300 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260619 04:48:43.625650 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.625183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55322)
0619 04:48:43.625413 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.625519 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260619 04:48:43.680543 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 18791
I20260619 04:48:43.697396 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 18438
I20260619 04:48:43.698509 18501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:48:43.813553 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 18438
I20260619 04:48:43.813582 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:43.813084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:33203 (local address 127.0.0.1:55336)
0619 04:48:43.813279 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:43.813403 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:33203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 04:48:43.835366 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42719
--webserver_interface=127.3.243.190
--webserver_port=41973
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:42719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:48:44.167933 18966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:44.168306 18966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:44.168419 18966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:44.175612 18966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:48:44.175729 18966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:44.175828 18966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:44.175927 18966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:48:44.175982 18966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:48:44.188838 18966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:42719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:42719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=41973
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:44.190164 18966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:44.191431 18966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:44.203606 18971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:44.203629 18972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:44.203563 18974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:44.204458 18966 server_base.cc:1034] running on GCE node
I20260619 04:48:44.205189 18966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:44.206547 18966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:44.207757 18966 hybrid_clock.cc:648] HybridClock initialized: now 1781844524207719 us; error 43 us; skew 500 ppm
I20260619 04:48:44.208209 18966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:44 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:44.211372 18966 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:48:44.212924 18966 webserver.cc:460] Webserver started at http://127.3.243.190:41973/ using document root <none> and password file <none>
I20260619 04:48:44.213599 18966 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:44.213747 18966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:44.219012 18966 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:48:44.221807 18981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:44.223143 18966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:48:44.223345 18966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "999fee5983b04f05a72050eba740fada"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "2093314be382603aa216b7162f489c69"
server_key_iv: "53eb8741a19a7b21bf7acf737e126f4f"
server_key_version: "encryptionkey@0"
I20260619 04:48:44.224365 18966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:44.242274 18966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:44.246129 18966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:44.246420 18966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:44.264830 18966 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:42719
I20260619 04:48:44.264849 19032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:42719 every 8 connection(s)
I20260619 04:48:44.266376 18966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:48:44.267843 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 18966
I20260619 04:48:44.268867 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:33203
--local_ip_for_outbound_sockets=127.3.243.129
--tserver_master_addrs=127.3.243.190:42719
--webserver_port=43021
--webserver_interface=127.3.243.129
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:48:44.276602 19033 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:48:44.280041 19033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Bootstrap starting.
I20260619 04:48:44.300087 19033 log.cc:824] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:44.315729 19033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:48:44.316360 19033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada: Bootstrap complete.
I20260619 04:48:44.325479 19033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:44.326399 19033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999fee5983b04f05a72050eba740fada, State: Initialized, Role: FOLLOWER
I20260619 04:48:44.327010 19033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:44.327328 19033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:48:44.327471 19033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:48:44.327673 19033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 2 FOLLOWER]: Advancing to term 3
I20260619 04:48:44.331501 19033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:44.331997 19033 leader_election.cc:302] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999fee5983b04f05a72050eba740fada; no voters:
I20260619 04:48:44.332698 19033 leader_election.cc:288] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 04:48:44.332836 19037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 3 FOLLOWER]: Leader election won for term 3
I20260619 04:48:44.333961 19037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [term 3 LEADER]: Becoming Leader. State: Replica: 999fee5983b04f05a72050eba740fada, State: Running, Role: LEADER
I20260619 04:48:44.334517 19037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } }
I20260619 04:48:44.334914 19033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:48:44.337600 19038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "999fee5983b04f05a72050eba740fada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } } }
I20260619 04:48:44.340315 19038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:44.337049 19039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999fee5983b04f05a72050eba740fada. Latest consensus state: current_term: 3 leader_uuid: "999fee5983b04f05a72050eba740fada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999fee5983b04f05a72050eba740fada" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 42719 } } }
I20260619 04:48:44.341265 19039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999fee5983b04f05a72050eba740fada [sys.catalog]: This master's current role is: LEADER
I20260619 04:48:44.345413 19043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:48:44.355331 19043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:48:44.356959 19043 catalog_manager.cc:1259] Loaded cluster ID: 5372fc6c60a843b9aaadf9d2304b265d
I20260619 04:48:44.357089 19043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 04:48:44.359251 19043 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd4d41bb9ad google::LogMessage::Fail() at ??:0
@ 0x7fd4d41be88a google::LogMessage::SendToLog() at ??:0
@ 0x7fd4d41bb4de google::LogMessage::Flush() at ??:0
@ 0x7fd4d41bca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd4e9bf2512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd4e9bef976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd4e9c77610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd4e9c77439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd4d6e07345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd4d6e0a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd4d6e0a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd4d6de5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd4da0bf6db start_thread at ??:0
@ 0x7fd4d305561f clone at ??:0
W20260619 04:48:44.663743 19035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:44.664078 19035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:44.664273 19035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:44.671531 19035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:44.671674 19035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:44.671851 19035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:48:44.681151 19035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:33203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=43021
--tserver_master_addrs=127.3.243.190:42719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:44.682446 19035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:44.683743 19035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:44.696894 19076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:44.696878 19074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:44.697047 19073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:44.697598 19035 server_base.cc:1034] running on GCE node
I20260619 04:48:44.698128 19035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:44.699012 19035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:44.700206 19035 hybrid_clock.cc:648] HybridClock initialized: now 1781844524700134 us; error 69 us; skew 500 ppm
I20260619 04:48:44.700525 19035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:44 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:44.703452 19035 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:48:44.704890 19035 webserver.cc:460] Webserver started at http://127.3.243.129:43021/ using document root <none> and password file <none>
I20260619 04:48:44.705502 19035 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:44.705595 19035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:44.710968 19035 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:48:44.713743 19083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:44.714982 19035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 04:48:44.715152 19035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "8545b9f0426a4b84bd02876f772754d3"
format_stamp: "Formatted at 2026-06-19 04:48:38 on dist-test-slave-2qp8"
server_key: "55cfbb922111ff19061733c7cc481457"
server_key_iv: "a0d529ce50316f1dbeeba8447af40a56"
server_key_version: "encryptionkey@0"
I20260619 04:48:44.716066 19035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:44.736181 19035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:44.740065 19035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:44.740381 19035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:44.741340 19035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:44.743108 19090 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260619 04:48:44.748621 19035 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 04:48:44.748749 19035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260619 04:48:44.748884 19035 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 04:48:44.751904 19035 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 04:48:44.752007 19035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260619 04:48:44.752416 19090 tablet_bootstrap.cc:492] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Bootstrap starting.
I20260619 04:48:44.779173 19090 log.cc:824] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Log is configured to *not* fsync() on all Append() calls
I20260619 04:48:44.790169 19035 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:33203
I20260619 04:48:44.790175 19197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:33203 every 8 connection(s)
I20260619 04:48:44.791776 19035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:48:44.794667 19090 tablet_bootstrap.cc:492] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:48:44.795392 19090 tablet_bootstrap.cc:492] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Bootstrap complete.
I20260619 04:48:44.795531 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 19035
I20260619 04:48:44.796407 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:38681
--local_ip_for_outbound_sockets=127.3.243.130
--tserver_master_addrs=127.3.243.190:42719
--webserver_port=45059
--webserver_interface=127.3.243.130
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:48:44.796579 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:44.794428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:42203)
0619 04:48:44.795265 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:44.795552 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":365,"threads_started":1}
I20260619 04:48:44.797158 19090 ts_tablet_manager.cc:1397] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Time spent bootstrapping tablet: real 0.045s user 0.025s sys 0.017s
W20260619 04:48:44.797272 19198 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:42719 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.190:42719: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
I20260619 04:48:44.798095 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:44.797763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:48915)
0619 04:48:44.797920 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:44.797994 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:48:44.798975 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:44.798635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:39813)
0619 04:48:44.798766 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:44.798851 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 04:48:44.799279 19198 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 04:48:44.806571 19090 raft_consensus.cc:348] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:44.807479 19090 raft_consensus.cc:729] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8545b9f0426a4b84bd02876f772754d3, State: Initialized, Role: FOLLOWER
I20260619 04:48:44.808076 19090 consensus_queue.cc:260] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:44.809628 19090 ts_tablet_manager.cc:1428] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260619 04:48:44.809815 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:44.809407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:41841)
0619 04:48:44.809577 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:44.809671 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 04:48:44.810796 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:44.810449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:33301)
0619 04:48:44.810573 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:44.810632 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 04:48:45.121006 19201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:45.121323 19201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:45.121487 19201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:45.128666 19201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:45.128821 19201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:45.128997 19201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:48:45.138548 19201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=45059
--tserver_master_addrs=127.3.243.190:42719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:45.140295 19201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:45.141566 19201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:45.154719 19207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:45.154752 19208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:45.154739 19210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:45.155508 19201 server_base.cc:1034] running on GCE node
I20260619 04:48:45.156044 19201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:45.156989 19201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:45.158221 19201 hybrid_clock.cc:648] HybridClock initialized: now 1781844525158181 us; error 52 us; skew 500 ppm
I20260619 04:48:45.158533 19201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:45 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:45.166085 19201 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:48:45.167670 19201 webserver.cc:460] Webserver started at http://127.3.243.130:45059/ using document root <none> and password file <none>
I20260619 04:48:45.168496 19201 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:45.168627 19201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:45.173981 19201 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:48:45.176669 19217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:45.178004 19201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:48:45.178185 19201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "9f1658cbafb44190845c63f6d446090c"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "431c5d0db143535e0e809daf19845b91"
server_key_iv: "256114625a1110da429b025be321fb20"
server_key_version: "encryptionkey@0"
I20260619 04:48:45.179159 19201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:45.197072 19201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:45.201399 19201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:45.201684 19201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:45.202699 19201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:45.204016 19201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:45.204146 19201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:45.204283 19201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:45.204384 19201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:45.237303 19201 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:38681
I20260619 04:48:45.237392 19329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:38681 every 8 connection(s)
I20260619 04:48:45.238750 19201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:48:45.242581 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 19201
I20260619 04:48:45.243225 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:40083
--local_ip_for_outbound_sockets=127.3.243.131
--tserver_master_addrs=127.3.243.190:42719
--webserver_port=36537
--webserver_interface=127.3.243.131
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:48:45.243575 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.241659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:35915)
0619 04:48:45.242385 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:45.242596 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":260,"threads_started":1}
W20260619 04:48:45.244320 19330 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:42719 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.190:42719: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
I20260619 04:48:45.245172 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.244858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:33051)
0619 04:48:45.244998 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:45.245074 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 04:48:45.246096 19332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.245736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:49467)
0619 04:48:45.245896 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:45.245981 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 04:48:45.246414 19330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:48:45.565510 19333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:48:45.565768 19333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:48:45.565945 19333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:48:45.573150 19333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:48:45.573253 19333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:48:45.573457 19333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:48:45.582664 19333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:40083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=36537
--tserver_master_addrs=127.3.243.190:42719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:48:45.583971 19333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:48:45.585321 19333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:48:45.599375 19338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:45.599397 19341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:48:45.599665 19339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:48:45.600159 19333 server_base.cc:1034] running on GCE node
I20260619 04:48:45.600766 19333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:48:45.601696 19333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:48:45.602896 19333 hybrid_clock.cc:648] HybridClock initialized: now 1781844525602847 us; error 41 us; skew 500 ppm
I20260619 04:48:45.603188 19333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:48:45 dist-test-slave-2qp8 krb5kdc[17883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:48:45.606455 19333 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:48:45.607933 19333 webserver.cc:460] Webserver started at http://127.3.243.131:36537/ using document root <none> and password file <none>
I20260619 04:48:45.608708 19333 fs_manager.cc:359] Metadata directory not provided
I20260619 04:48:45.608812 19333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:48:45.614332 19333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 04:48:45.617117 19348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:45.618463 19333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:48:45.618594 19333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "01205c6127cc44bcbbffe4168838e233"
format_stamp: "Formatted at 2026-06-19 04:48:39 on dist-test-slave-2qp8"
server_key: "a6db3955180c1edd91392a0870d277ee"
server_key_iv: "4fa463a5998b0ff67862c3558f9bff6e"
server_key_version: "encryptionkey@0"
I20260619 04:48:45.619693 19333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:48:45.638751 19333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:48:45.642692 19333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:48:45.642916 19333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:48:45.643803 19333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:48:45.645083 19333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:48:45.645171 19333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:45.645260 19333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:48:45.645308 19333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:48:45.677785 19333 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:40083
I20260619 04:48:45.677874 19460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:40083 every 8 connection(s)
I20260619 04:48:45.679304 19333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:48:45.684163 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.682164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:40385)
0619 04:48:45.682937 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:45.683162 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":278,"threads_started":1}
W20260619 04:48:45.684813 19461 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:42719 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.190:42719: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
I20260619 04:48:45.685586 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.685280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:38021)
0619 04:48:45.685412 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:45.685474 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 04:48:45.686399 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.686097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:45209)
0619 04:48:45.686231 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:45.686299 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 04:48:45.686599 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 19333
W20260619 04:48:45.686661 19461 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 04:48:45.812450 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.811553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:55159)
0619 04:48:45.812056 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:45.812248 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":262,"threads_started":1}
I20260619 04:48:45.971809 19466 raft_consensus.cc:482] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:45.972146 19466 raft_consensus.cc:504] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:45.973773 19466 leader_election.cc:288] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681), 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
Jun 19 04:48:45 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
Jun 19 04:48:45 dist-test-slave-2qp8 krb5kdc[17883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:48:45.995560 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.974187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:40083 (local address 127.3.243.129:52865)
0619 04:48:45.977511 (+ 3324us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:45.977529 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:45.977670 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:45.979879 (+ 2209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:45.979891 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:45.980719 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:45.981416 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:45.981435 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:45.983017 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:45.983021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:45.984507 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:45.984519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:45.985456 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:45.985463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:45.985711 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:45.986563 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:45.986580 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:45.988881 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:45.992241 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:45.992249 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:45.992262 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:45.992680 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:45.993111 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:45.993116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:45.993119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:45.993324 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:45.994072 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:45.994079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:45.994381 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 04:48:45.994657 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 04:48:45.995102 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3210,"mutex_wait_us":184,"spinlock_wait_cycles":27008,"thread_start_us":3074,"threads_started":1}
I20260619 04:48:45.995644 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.974504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:52865 (local address 127.3.243.131:40083)
0619 04:48:45.976877 (+ 2373us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:45.976898 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:45.977629 (+ 731us) server_negotiation.cc:407] Connection header received
0619 04:48:45.979103 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:45.979110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:45.979421 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:45.979775 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:45.981601 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:45.982831 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:45.984651 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:45.985245 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:45.985433 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:45.989164 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:45.989171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:45.989184 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:45.989223 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:45.992047 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:45.992826 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:45.992832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:45.992837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:45.992985 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:45.993445 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:45.993451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:45.993459 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:45.993924 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:45.994110 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:45.994880 (+ 770us) server_negotiation.cc:299] Negotiation successful
0619 04:48:45.995266 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2136,"thread_start_us":1920,"threads_started":1}
I20260619 04:48:45.995972 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.974047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:38681 (local address 127.3.243.129:51597)
0619 04:48:45.974365 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:45.974428 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 04:48:45.975126 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:48:45.979879 (+ 4753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:48:45.979892 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:48:45.980719 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:48:45.981416 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:45.981434 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:45.983045 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:45.983050 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:45.984514 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:48:45.984526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:48:45.985521 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:48:45.985528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:48:45.985713 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:45.986496 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:48:45.986520 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:48:45.989320 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:48:45.993101 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:45.993108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:45.993113 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:45.993510 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:45.993932 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:48:45.993937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:48:45.993941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:48:45.994067 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:48:45.994739 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:48:45.994744 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:48:45.994879 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 04:48:45.995028 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 04:48:45.995283 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":84}
I20260619 04:48:45.996052 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:45.974204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:51597 (local address 127.3.243.130:38681)
0619 04:48:45.977530 (+ 3326us) server_negotiation.cc:206] Beginning negotiation
0619 04:48:45.977540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:48:45.977570 (+ 30us) server_negotiation.cc:407] Connection header received
0619 04:48:45.979045 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:48:45.979053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:48:45.979421 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:48:45.979773 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:48:45.981613 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:45.982889 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:45.984653 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:48:45.985255 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:48:45.985451 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:48:45.989556 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:48:45.989563 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:48:45.989574 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:48:45.989613 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:48:45.992973 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:45.993650 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:45.993657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:45.993662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:45.993813 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:48:45.994182 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:48:45.994191 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:48:45.994193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:48:45.994612 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:48:45.994809 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 04:48:45.995261 (+ 452us) server_negotiation.cc:299] Negotiation successful
0619 04:48:45.995632 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3174,"thread_start_us":3058,"threads_started":1}
W20260619 04:48:45.998829 19084 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
W20260619 04:48:45.999729 19087 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
I20260619 04:48:45.999852 19087 leader_election.cc:302] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8545b9f0426a4b84bd02876f772754d3; no voters: 01205c6127cc44bcbbffe4168838e233, 9f1658cbafb44190845c63f6d446090c
I20260619 04:48:46.000375 19466 raft_consensus.cc:2726] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:48:46.247753 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:46.246909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:42729)
0619 04:48:46.247385 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:46.247531 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":217,"threads_started":1}
I20260619 04:48:46.687839 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:46.687143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:46253)
0619 04:48:46.687534 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:46.687640 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":190,"threads_started":1}
I20260619 04:48:46.814162 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:46.813322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:47477)
0619 04:48:46.813779 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:46.813909 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260619 04:48:47.249301 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:47.248654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:53757)
0619 04:48:47.249064 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:47.249141 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":220,"threads_started":1}
I20260619 04:48:47.689334 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:47.688667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:54639)
0619 04:48:47.689074 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:47.689148 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":206,"threads_started":1}
I20260619 04:48:47.815924 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:47.815164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:56373)
0619 04:48:47.815660 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:47.815741 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":240,"threads_started":1}
I20260619 04:48:48.117106 19476 raft_consensus.cc:482] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:48.117439 19476 raft_consensus.cc:504] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:48.118120 19476 leader_election.cc:288] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681), 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
W20260619 04:48:48.119099 19084 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
W20260619 04:48:48.119302 19087 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
I20260619 04:48:48.119434 19087 leader_election.cc:302] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8545b9f0426a4b84bd02876f772754d3; no voters: 01205c6127cc44bcbbffe4168838e233, 9f1658cbafb44190845c63f6d446090c
I20260619 04:48:48.119675 19476 raft_consensus.cc:2726] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:48:48.250825 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:48.250162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:36787)
0619 04:48:48.250593 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:48.250669 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20260619 04:48:48.690725 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:48.690110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:44723)
0619 04:48:48.690508 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:48.690584 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":228,"threads_started":1}
I20260619 04:48:48.817596 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:48.816890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:36679)
0619 04:48:48.817349 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:48.817434 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260619 04:48:49.252655 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:49.251820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:44303)
0619 04:48:49.252349 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:49.252463 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":312,"threads_started":1}
I20260619 04:48:49.692478 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:49.691672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:56273)
0619 04:48:49.692167 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:49.692292 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":232,"threads_started":1}
I20260619 04:48:49.819413 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:49.818666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:48249)
0619 04:48:49.819124 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:49.819229 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":229,"threads_started":1}
I20260619 04:48:50.254372 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:50.253672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:48689)
0619 04:48:50.254096 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:50.254178 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":231,"threads_started":1}
I20260619 04:48:50.543352 19484 raft_consensus.cc:482] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:50.543570 19484 raft_consensus.cc:504] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:50.544273 19484 leader_election.cc:288] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681), 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
W20260619 04:48:50.545164 19084 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
W20260619 04:48:50.545377 19087 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
I20260619 04:48:50.545508 19087 leader_election.cc:302] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8545b9f0426a4b84bd02876f772754d3; no voters: 01205c6127cc44bcbbffe4168838e233, 9f1658cbafb44190845c63f6d446090c
I20260619 04:48:50.545739 19484 raft_consensus.cc:2726] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:48:50.693984 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:50.693364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:34705)
0619 04:48:50.693755 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:50.693837 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":209,"threads_started":1}
I20260619 04:48:50.821105 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:50.820288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:35501)
0619 04:48:50.820803 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:50.820914 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":211,"threads_started":1}
I20260619 04:48:51.256047 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:51.255201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:47937)
0619 04:48:51.255794 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:51.255873 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":301,"threads_started":1}
I20260619 04:48:51.695664 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:51.694834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:57793)
0619 04:48:51.695365 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:51.695456 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":288,"threads_started":1}
I20260619 04:48:51.822844 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:51.821976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:40087)
0619 04:48:51.822508 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:51.822598 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":266,"threads_started":1}
I20260619 04:48:52.257874 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:52.257070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:53113)
0619 04:48:52.257605 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:52.257692 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":264,"threads_started":1}
I20260619 04:48:52.697436 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:52.696639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:34391)
0619 04:48:52.697127 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:52.697209 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":226,"threads_started":1}
I20260619 04:48:52.824604 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:52.823764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:53937)
0619 04:48:52.824354 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:52.824440 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":296,"threads_started":1}
I20260619 04:48:53.259497 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:53.258825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:58913)
0619 04:48:53.259230 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:53.259308 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":201,"threads_started":1}
I20260619 04:48:53.699108 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:53.698379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:33793)
0619 04:48:53.698823 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:53.698924 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":235,"threads_started":1}
I20260619 04:48:53.826328 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:53.825556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:43381)
0619 04:48:53.826021 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:53.826113 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":239,"threads_started":1}
I20260619 04:48:54.261168 19496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:54.260471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:56471)
0619 04:48:54.260933 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:54.261013 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":206,"threads_started":1}
I20260619 04:48:54.530966 19497 raft_consensus.cc:482] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:54.531144 19497 raft_consensus.cc:504] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:54.531766 19497 leader_election.cc:288] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681), 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
W20260619 04:48:54.532712 19084 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
W20260619 04:48:54.532964 19087 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
I20260619 04:48:54.533102 19087 leader_election.cc:302] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8545b9f0426a4b84bd02876f772754d3; no voters: 01205c6127cc44bcbbffe4168838e233, 9f1658cbafb44190845c63f6d446090c
I20260619 04:48:54.533365 19497 raft_consensus.cc:2726] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:48:54.701033 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:54.700227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:45501)
0619 04:48:54.700794 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:54.700880 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":257,"threads_started":1}
I20260619 04:48:54.828258 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:54.827260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:51841)
0619 04:48:54.827828 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:54.827956 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":308,"threads_started":1}
I20260619 04:48:55.262785 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:55.262106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:51013)
0619 04:48:55.262511 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:55.262591 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":211,"threads_started":1}
I20260619 04:48:55.702667 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:55.701959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:60077)
0619 04:48:55.702404 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:55.702490 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":226,"threads_started":1}
I20260619 04:48:55.829825 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:55.829079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:46625)
0619 04:48:55.829542 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:55.829620 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":241,"threads_started":1}
I20260619 04:48:56.264608 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:56.263831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:60865)
0619 04:48:56.264366 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:56.264452 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":235,"threads_started":1}
I20260619 04:48:56.704550 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:56.703673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:58291)
0619 04:48:56.704275 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:56.704368 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":246,"threads_started":1}
I20260619 04:48:56.831408 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:56.830692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:38751)
0619 04:48:56.831129 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:56.831208 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260619 04:48:57.266311 19506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:57.265519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:43071)
0619 04:48:57.266068 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:57.266148 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":335,"threads_started":1}
I20260619 04:48:57.706223 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:57.705522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:59621)
0619 04:48:57.705982 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:57.706058 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":249,"threads_started":1}
I20260619 04:48:57.833130 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:57.832328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:34011)
0619 04:48:57.832837 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:57.832915 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":275,"threads_started":1}
I20260619 04:48:57.833353 19509 raft_consensus.cc:482] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:48:57.833577 19509 raft_consensus.cc:504] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f1658cbafb44190845c63f6d446090c" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 38681 } } peers { permanent_uuid: "8545b9f0426a4b84bd02876f772754d3" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 33203 } } peers { permanent_uuid: "01205c6127cc44bcbbffe4168838e233" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 40083 } }
I20260619 04:48:57.834118 19509 leader_election.cc:288] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681), 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083)
W20260619 04:48:57.835058 19084 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 01205c6127cc44bcbbffe4168838e233 (127.3.243.131:40083): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
W20260619 04:48:57.835238 19087 leader_election.cc:341] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f1658cbafb44190845c63f6d446090c (127.3.243.130:38681): Not found: Tablet not found: 2b7f8acf1995490f84fe79abc57443c9
I20260619 04:48:57.835368 19087 leader_election.cc:302] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8545b9f0426a4b84bd02876f772754d3; no voters: 01205c6127cc44bcbbffe4168838e233, 9f1658cbafb44190845c63f6d446090c
I20260619 04:48:57.835642 19509 raft_consensus.cc:2726] T 2b7f8acf1995490f84fe79abc57443c9 P 8545b9f0426a4b84bd02876f772754d3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 04:48:58.267971 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:58.267225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:49563)
0619 04:48:58.267695 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:58.267793 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":244,"threads_started":1}
I20260619 04:48:58.707828 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:58.707093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:39873)
0619 04:48:58.707576 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:58.707665 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":266,"threads_started":1}
I20260619 04:48:58.834892 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:58.834129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:53341)
0619 04:48:58.834625 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:58.834715 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":302,"threads_started":1}
I20260619 04:48:59.269670 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:59.268913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:53727)
0619 04:48:59.269394 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:59.269474 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260619 04:48:59.709307 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:59.708608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:56415)
0619 04:48:59.709042 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:59.709121 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260619 04:48:59.836884 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:48:59.836168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.129:45229)
0619 04:48:59.836618 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0619 04:48:59.836702 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260619 04:49:00.271760 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:00.270930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.130:46021)
0619 04:49:00.271481 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:00.271600 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":307,"threads_started":1}
I20260619 04:49:00.691654 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 19035
I20260619 04:49:00.708377 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:00.692722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.3.243.129:33203)
0619 04:49:00.693259 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:00.693266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:00.693333 (+ 67us) server_negotiation.cc:407] Connection header received
0619 04:49:00.693407 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:00.693415 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:00.693722 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:00.693891 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:00.696914 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:00.697867 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:00.699189 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:00.699755 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:00.699861 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:00.702582 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:00.702596 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:00.702615 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:00.702689 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:00.705499 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:00.706176 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:00.706189 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:00.706191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:00.706331 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:00.706745 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:00.706751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:00.706754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:00.707111 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:00.707435 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:00.707732 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 04:49:00.708022 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":223,"threads_started":1}
I20260619 04:49:00.709589 19193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:00.720872 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:00.710406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:42719 (local address 127.3.243.131:53025)
0619 04:49:00.720566 (+ 10160us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:00.720715 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:42719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10027,"thread_start_us":9897,"threads_started":1}
I20260619 04:49:00.800649 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 19035
I20260619 04:49:00.818651 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 19201
I20260619 04:49:00.832692 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:00.819459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.3.243.130:38681)
0619 04:49:00.819981 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:00.819985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:00.820001 (+ 16us) server_negotiation.cc:407] Connection header received
0619 04:49:00.820055 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:00.820058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:00.820166 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:00.820275 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:00.821730 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:00.822509 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:00.823956 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:00.824523 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:00.824616 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:00.827373 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:00.827377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:00.827380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:00.827412 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:00.830137 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:00.830788 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:00.830792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:00.830794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:00.830902 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:00.831293 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:00.831296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:00.831298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:00.831546 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:00.831741 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:00.831995 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 04:49:00.832395 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":291,"threads_started":1}
I20260619 04:49:00.833452 19325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:00.917403 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 19201
I20260619 04:49:00.935073 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 19333
I20260619 04:49:00.948747 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:00.935956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.3.243.131:40083)
0619 04:49:00.936613 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:00.936618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:00.936635 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:49:00.936692 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:00.936695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:00.936762 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:00.936857 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:00.938289 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:00.939247 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:00.940482 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:00.940883 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:00.940978 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:00.943568 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:00.943573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:00.943576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:00.943606 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:00.946376 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:00.947008 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:00.947012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:00.947014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:00.947141 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:00.947435 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:00.947438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:00.947440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:00.947659 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:00.947860 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:00.948048 (+ 188us) server_negotiation.cc:299] Negotiation successful
0619 04:49:00.948467 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":407,"threads_started":1}
I20260619 04:49:00.949486 19456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:01.032775 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 19333
2026-06-19T04:49:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24435 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:49:01 dist-test-slave-2qp8 krb5kdc[19526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:49:01 dist-test-slave-2qp8 krb5kdc[19526](info): set up 2 sockets
Jun 19 04:49:01 dist-test-slave-2qp8 krb5kdc[19526](info): commencing operation
krb5kdc: starting...
W20260619 04:49:02.144953 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:49:02 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:49:02.384613 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44279
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:49:02.713953 19539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:02.714318 19539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:02.714473 19539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:02.721678 19539 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 04:49:02.721824 19539 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 04:49:02.721920 19539 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 04:49:02.722016 19539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:49:02.722117 19539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:02.722230 19539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:02.722319 19539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:49:02.722400 19539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:49:02.732255 19539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:02.733529 19539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:02.734877 19539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:02.746474 19544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:02.746553 19545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:02.746697 19547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:02.747454 19539 server_base.cc:1034] running on GCE node
I20260619 04:49:02.748250 19539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:02.749549 19539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:02.750769 19539 hybrid_clock.cc:648] HybridClock initialized: now 1781844542750716 us; error 36 us; skew 500 ppm
I20260619 04:49:02.751121 19539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:02 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844542, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:02.759701 19539 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:49:02.761150 19539 webserver.cc:460] Webserver started at http://127.3.243.190:35409/ using document root <none> and password file <none>
I20260619 04:49:02.761799 19539 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:02.761883 19539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:02.762144 19539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:49:02.765118 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "5a56f0f1b0a14868b6769224b8c56255"
format_stamp: "Formatted at 2026-06-19 04:49:02 on dist-test-slave-2qp8"
server_key: "b328a5d2c1d6519708d5183d49b6d6d9"
server_key_iv: "c961c9a0d2e067807a262e4797f9b4b7"
server_key_version: "encryptionkey@0"
I20260619 04:49:02.765823 19539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "5a56f0f1b0a14868b6769224b8c56255"
format_stamp: "Formatted at 2026-06-19 04:49:02 on dist-test-slave-2qp8"
server_key: "b328a5d2c1d6519708d5183d49b6d6d9"
server_key_iv: "c961c9a0d2e067807a262e4797f9b4b7"
server_key_version: "encryptionkey@0"
I20260619 04:49:02.770655 19539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 04:49:02.773913 19554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:02.775177 19539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 04:49:02.775360 19539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "5a56f0f1b0a14868b6769224b8c56255"
format_stamp: "Formatted at 2026-06-19 04:49:02 on dist-test-slave-2qp8"
server_key: "b328a5d2c1d6519708d5183d49b6d6d9"
server_key_iv: "c961c9a0d2e067807a262e4797f9b4b7"
server_key_version: "encryptionkey@0"
I20260619 04:49:02.775552 19539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:02.820613 19539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:02.825242 19539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:02.825547 19539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:02.844205 19539 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44279
I20260619 04:49:02.844233 19605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44279 every 8 connection(s)
I20260619 04:49:02.845674 19539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:49:02.847872 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 19539
I20260619 04:49:02.848081 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:49:02.849026 4046 external_mini_cluster.cc:1428] Setting key 99028ff8ebfc7bbd22ff3217639cfcf3
I20260619 04:49:02.850117 19606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:49:02 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:49:02.864815 19606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Bootstrap starting.
I20260619 04:49:02.868283 19606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Neither blocks nor log segments found. Creating new log.
I20260619 04:49:02.869530 19606 log.cc:824] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Log is configured to *not* fsync() on all Append() calls
I20260619 04:49:02.872577 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:02.851082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.3.243.190:44279)
0619 04:49:02.851869 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:02.851880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:02.851920 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:49:02.853469 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:02.853489 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:02.853874 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:02.854186 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:02.855893 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:02.857311 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:02.859126 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:02.859752 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:02.859896 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:02.864972 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:02.864981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:02.864995 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:02.865034 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:02.867926 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:02.868808 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:02.868814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:02.868820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:02.868985 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:02.869404 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:02.869408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:02.869409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:02.870010 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:02.870343 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:02.870758 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 04:49:02.871231 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":358,"threads_started":1}
I20260619 04:49:02.872946 19606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: No bootstrap required, opened a new log
I20260619 04:49:02.881219 19606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:02.881567 19606 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:49:02.881644 19606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a56f0f1b0a14868b6769224b8c56255, State: Initialized, Role: FOLLOWER
I20260619 04:49:02.882122 19606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:02.882318 19606 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:49:02.882413 19606 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:49:02.882537 19606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:49:02.884774 19606 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:02.885358 19606 leader_election.cc:302] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a56f0f1b0a14868b6769224b8c56255; no voters:
I20260619 04:49:02.885916 19606 leader_election.cc:288] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:49:02.886051 19611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:49:02.886778 19611 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 1 LEADER]: Becoming Leader. State: Replica: 5a56f0f1b0a14868b6769224b8c56255, State: Running, Role: LEADER
I20260619 04:49:02.887298 19611 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:02.888002 19606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:49:02.890905 19612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a56f0f1b0a14868b6769224b8c56255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } } }
I20260619 04:49:02.892066 19612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: This master's current role is: LEADER
I20260619 04:49:02.891906 19613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a56f0f1b0a14868b6769224b8c56255. Latest consensus state: current_term: 1 leader_uuid: "5a56f0f1b0a14868b6769224b8c56255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } } }
I20260619 04:49:02.892623 19613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: This master's current role is: LEADER
I20260619 04:49:02.897352 19620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:49:02.902696 19620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:49:02.911523 19620 catalog_manager.cc:1347] Generated new cluster ID: 2d31ef88f21348308fce532e483568e7
I20260619 04:49:02.911643 19620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:49:02.952992 19620 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:49:02.954326 19620 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:49:02.962064 19620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Generated new TSK 0
I20260619 04:49:02.962932 19620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:49:03.024343 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44279
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:49:03.359220 19632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:03.359534 19632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:03.359730 19632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:03.367080 19632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:03.367251 19632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:03.367445 19632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:49:03.377530 19632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:03.378960 19632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:03.380471 19632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:03.393996 19637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:03.394071 19638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:03.394065 19640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:03.394917 19632 server_base.cc:1034] running on GCE node
I20260619 04:49:03.395475 19632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:03.396420 19632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:03.397645 19632 hybrid_clock.cc:648] HybridClock initialized: now 1781844543397581 us; error 57 us; skew 500 ppm
I20260619 04:49:03.397966 19632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:03 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844543, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:03.401019 19632 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:49:03.402515 19632 webserver.cc:460] Webserver started at http://127.3.243.129:43141/ using document root <none> and password file <none>
I20260619 04:49:03.403170 19632 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:03.403265 19632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:03.403506 19632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:49:03.406404 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "6974b0ea51884bed912d50e8209023e9"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6a336f8632e1f25ba03394c5a44fbb5e"
server_key_iv: "d984e87562b13962c3a2f5782a72aeb3"
server_key_version: "encryptionkey@0"
I20260619 04:49:03.407076 19632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "6974b0ea51884bed912d50e8209023e9"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6a336f8632e1f25ba03394c5a44fbb5e"
server_key_iv: "d984e87562b13962c3a2f5782a72aeb3"
server_key_version: "encryptionkey@0"
I20260619 04:49:03.412045 19632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 04:49:03.415400 19647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:03.416688 19632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:49:03.416863 19632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "6974b0ea51884bed912d50e8209023e9"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6a336f8632e1f25ba03394c5a44fbb5e"
server_key_iv: "d984e87562b13962c3a2f5782a72aeb3"
server_key_version: "encryptionkey@0"
I20260619 04:49:03.417042 19632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:03.434181 19632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:03.437978 19632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:03.438254 19632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:03.439198 19632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:49:03.440712 19632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:49:03.440793 19632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:03.440963 19632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:49:03.441028 19632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:03.473970 19632 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:43111
I20260619 04:49:03.474014 19759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:43111 every 8 connection(s)
I20260619 04:49:03.475276 19632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:49:03.481446 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 19632
I20260619 04:49:03.481671 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:49:03.482528 4046 external_mini_cluster.cc:1428] Setting key 401945ac18cbd8718a19beef8e659174
Jun 19 04:49:03 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844543, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:49:03.497038 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:03.477700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:47909 (local address 127.3.243.190:44279)
0619 04:49:03.478163 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:03.478168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:03.479034 (+ 866us) server_negotiation.cc:407] Connection header received
0619 04:49:03.481101 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:03.481106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:03.481175 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:03.481267 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:03.483357 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:03.484410 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:03.486831 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:03.487204 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:03.487296 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:03.491413 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:03.491418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:03.491421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:03.491451 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:03.494050 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:03.494736 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:03.494744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:03.494749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:03.494868 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:03.495264 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:03.495271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:03.495274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:03.495538 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:03.495705 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:03.496464 (+ 759us) server_negotiation.cc:299] Negotiation successful
0619 04:49:03.496682 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":227,"threads_started":1}
I20260619 04:49:03.497957 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:03.478157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:47909)
0619 04:49:03.478877 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:03.478913 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:03.480813 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:03.481564 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:03.481577 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:03.482375 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:49:03.483185 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:03.483205 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:03.484581 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:03.484587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:03.486683 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:03.486699 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:03.487421 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:03.487430 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:03.487764 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:03.488548 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:49:03.488587 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:49:03.491169 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:49:03.494185 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:03.494194 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:03.494210 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:03.494620 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:03.494975 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:03.494980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:03.494984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:03.495159 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:03.495691 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:49:03.495699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:49:03.496029 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 04:49:03.496373 (+ 344us) client_negotiation.cc:240] Negotiation successful
0619 04:49:03.496885 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":323,"threads_started":1}
I20260619 04:49:03.499459 19760 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44279
I20260619 04:49:03.499797 19760 heartbeater.cc:464] Registering TS with master...
I20260619 04:49:03.500641 19760 heartbeater.cc:511] Master 127.3.243.190:44279 requested a full tablet report, sending...
I20260619 04:49:03.502821 19571 ts_manager.cc:194] Registered new tserver with Master: 6974b0ea51884bed912d50e8209023e9 (127.3.243.129:43111)
I20260619 04:49:03.504366 19571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:47909
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:49:03.537132 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44279
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:49:03.866541 19766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:03.866873 19766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:03.867008 19766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:03.874596 19766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:03.874709 19766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:03.874822 19766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:49:03.884775 19766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:03.885968 19766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:03.887182 19766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:03.899971 19771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:03.900002 19772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:03.900337 19774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:03.901142 19766 server_base.cc:1034] running on GCE node
I20260619 04:49:03.901818 19766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:03.902753 19766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:03.903967 19766 hybrid_clock.cc:648] HybridClock initialized: now 1781844543903900 us; error 60 us; skew 500 ppm
I20260619 04:49:03.904310 19766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:03 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844543, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:03.907325 19766 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:49:03.908875 19766 webserver.cc:460] Webserver started at http://127.3.243.130:40479/ using document root <none> and password file <none>
I20260619 04:49:03.909572 19766 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:03.909672 19766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:03.909957 19766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:49:03.912966 19766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "d775e9fd77ca487d9a05960d898ac193"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6dea764294db77aef248098aa2aa7c57"
server_key_iv: "b5a7c44e804515c5d964fa4c3c85d9a5"
server_key_version: "encryptionkey@0"
I20260619 04:49:03.913628 19766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "d775e9fd77ca487d9a05960d898ac193"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6dea764294db77aef248098aa2aa7c57"
server_key_iv: "b5a7c44e804515c5d964fa4c3c85d9a5"
server_key_version: "encryptionkey@0"
I20260619 04:49:03.918658 19766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:49:03.922199 19781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:03.923483 19766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:49:03.923651 19766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "d775e9fd77ca487d9a05960d898ac193"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6dea764294db77aef248098aa2aa7c57"
server_key_iv: "b5a7c44e804515c5d964fa4c3c85d9a5"
server_key_version: "encryptionkey@0"
I20260619 04:49:03.923848 19766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:03.938315 19766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:03.942188 19766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:03.942467 19766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:03.943527 19766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:49:03.944870 19766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:49:03.944938 19766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:03.945101 19766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:49:03.945154 19766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:03.977316 19766 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:42135
I20260619 04:49:03.977361 19893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:42135 every 8 connection(s)
I20260619 04:49:03.978668 19766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:49:03.979693 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 19766
I20260619 04:49:03.979883 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:49:03.980536 4046 external_mini_cluster.cc:1428] Setting key 47c05c68bef15d84d86223a08880567d
Jun 19 04:49:03 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844543, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:49:04.001909 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:03.981726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:45943 (local address 127.3.243.190:44279)
0619 04:49:03.981891 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:03.981895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:03.983507 (+ 1612us) server_negotiation.cc:407] Connection header received
0619 04:49:03.986413 (+ 2906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:03.986418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:03.986485 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:03.986568 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:03.988741 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:03.989672 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:03.992201 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:03.992635 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:03.992774 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:03.996197 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:03.996202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:03.996205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:03.996244 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:03.998805 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:03.999526 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:03.999535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:03.999537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:03.999661 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:04.000132 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:04.000140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:04.000144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:04.000432 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:04.000614 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.001309 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.001543 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 04:49:04.002911 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:03.982397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:45943)
0619 04:49:03.983307 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:03.983369 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:03.986119 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:03.986846 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:03.986859 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:03.987633 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:49:03.988534 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:03.988564 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:03.989841 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:03.989849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:03.992026 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:03.992039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:03.992869 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:03.992877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:03.993110 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:03.993818 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:49:03.993840 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:49:03.995932 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:49:03.998945 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:03.998954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:03.998972 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:03.999400 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:03.999777 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:03.999783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:03.999787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:03.999990 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:04.000574 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:49:04.000584 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:49:04.000941 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 04:49:04.001204 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 04:49:04.001758 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":367,"threads_started":1}
I20260619 04:49:04.004470 19894 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44279
I20260619 04:49:04.004782 19894 heartbeater.cc:464] Registering TS with master...
I20260619 04:49:04.005491 19894 heartbeater.cc:511] Master 127.3.243.190:44279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:49:04.007006 19571 ts_manager.cc:194] Registered new tserver with Master: d775e9fd77ca487d9a05960d898ac193 (127.3.243.130:42135)
I20260619 04:49:04.007758 19571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:45943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:49:04.037539 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44279
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:49:04.366773 19899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:04.367059 19899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:04.367153 19899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:04.374531 19899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:04.374650 19899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:04.374765 19899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:49:04.384651 19899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:04.385949 19899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:04.387295 19899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:04.400676 19904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:04.400841 19907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:04.400846 19905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:04.402087 19899 server_base.cc:1034] running on GCE node
I20260619 04:49:04.402920 19899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:04.403925 19899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:04.405154 19899 hybrid_clock.cc:648] HybridClock initialized: now 1781844544405121 us; error 49 us; skew 500 ppm
I20260619 04:49:04.405488 19899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:04 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:04.408762 19899 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:49:04.410295 19899 webserver.cc:460] Webserver started at http://127.3.243.131:45225/ using document root <none> and password file <none>
I20260619 04:49:04.410989 19899 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:04.411152 19899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:04.411474 19899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:49:04.414597 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "8d43d16d595448b98dd92aec65a41f4e"
format_stamp: "Formatted at 2026-06-19 04:49:04 on dist-test-slave-2qp8"
server_key: "bec03d1e5ab74c60eeb2c34c4497d97c"
server_key_iv: "9de03abf772ae98e686dd826b4cb3ac1"
server_key_version: "encryptionkey@0"
I20260619 04:49:04.415359 19899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "8d43d16d595448b98dd92aec65a41f4e"
format_stamp: "Formatted at 2026-06-19 04:49:04 on dist-test-slave-2qp8"
server_key: "bec03d1e5ab74c60eeb2c34c4497d97c"
server_key_iv: "9de03abf772ae98e686dd826b4cb3ac1"
server_key_version: "encryptionkey@0"
I20260619 04:49:04.420583 19899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 04:49:04.424292 19914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:04.425635 19899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:49:04.425825 19899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "8d43d16d595448b98dd92aec65a41f4e"
format_stamp: "Formatted at 2026-06-19 04:49:04 on dist-test-slave-2qp8"
server_key: "bec03d1e5ab74c60eeb2c34c4497d97c"
server_key_iv: "9de03abf772ae98e686dd826b4cb3ac1"
server_key_version: "encryptionkey@0"
I20260619 04:49:04.426033 19899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:04.439222 19899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:04.443549 19899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:04.443863 19899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:04.444929 19899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:49:04.446261 19899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:49:04.446360 19899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:04.446507 19899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:49:04.446584 19899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:04.482095 19899 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:44585
I20260619 04:49:04.482131 20026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:44585 every 8 connection(s)
I20260619 04:49:04.483374 19899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:49:04.491350 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 19899
I20260619 04:49:04.491571 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:49:04.492154 4046 external_mini_cluster.cc:1428] Setting key 94ea1734709d664ac498e9666ebdf356
Jun 19 04:49:04 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844544, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:49:04.506453 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.486010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:56601 (local address 127.3.243.190:44279)
0619 04:49:04.486235 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.486240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.487387 (+ 1147us) server_negotiation.cc:407] Connection header received
0619 04:49:04.489622 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.489628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.489726 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.489855 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:04.491757 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.492964 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.495206 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.495859 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.496018 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.500453 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:04.500459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:04.500463 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:04.500516 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:04.503081 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:04.503918 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:04.503926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:04.503930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:04.504076 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:04.504594 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:04.504602 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:04.504605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:04.504904 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:04.505088 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.505782 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.506035 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:49:04.507763 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.486475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:56601)
0619 04:49:04.487218 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:04.487265 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:04.489293 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:04.490104 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:04.490114 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:04.490678 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:49:04.491560 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.491591 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.493151 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.493159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.495001 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.495017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.496138 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.496148 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.496496 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.497454 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:49:04.497487 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:49:04.500230 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:49:04.503243 (+ 3013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:04.503253 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:04.503273 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:04.503775 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:04.504258 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:04.504264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:04.504268 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:04.504473 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:04.505045 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:49:04.505054 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:49:04.505414 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 04:49:04.505687 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 04:49:04.506256 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":289,"threads_started":1}
I20260619 04:49:04.507926 19760 heartbeater.cc:503] Master 127.3.243.190:44279 was elected leader, sending a full tablet report...
I20260619 04:49:04.509181 20027 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44279
I20260619 04:49:04.509656 20027 heartbeater.cc:464] Registering TS with master...
I20260619 04:49:04.510470 20027 heartbeater.cc:511] Master 127.3.243.190:44279 requested a full tablet report, sending...
I20260619 04:49:04.511878 19570 ts_manager.cc:194] Registered new tserver with Master: 8d43d16d595448b98dd92aec65a41f4e (127.3.243.131:44585)
I20260619 04:49:04.513039 19570 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:56601
I20260619 04:49:04.520270 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:49:04.538453 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.525081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.3.243.190:44279)
0619 04:49:04.525396 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.525403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.525690 (+ 287us) server_negotiation.cc:407] Connection header received
0619 04:49:04.525907 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.525911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.525973 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.526079 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:04.527612 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.528449 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.529973 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.530406 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.530538 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.532994 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:04.533000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:04.533003 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:04.533034 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:04.535643 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:04.536398 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:04.536403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:04.536407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:04.536537 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:04.536920 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:04.536947 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:04.536950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:04.537299 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:04.537556 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.537881 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.538107 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 04:49:04.542827 19570 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:49:04.550632 19570 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:49:04.574329 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.563437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:42135 (local address 127.0.0.1:55414)
0619 04:49:04.564898 (+ 1461us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:04.564941 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:04.565096 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:04.566059 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:04.566066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:04.566096 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:04.566508 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.566527 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.568295 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.568302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.570708 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.570725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.573116 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.573125 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.573437 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.573517 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 04:49:04.573651 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 04:49:04.573897 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"thread_start_us":5242,"threads_started":1}
I20260619 04:49:04.575835 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.563464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55414 (local address 127.3.243.130:42135)
0619 04:49:04.565567 (+ 2103us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.565574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.565595 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:49:04.565669 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.565675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.565860 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.566041 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:04.567073 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.568077 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.570924 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.572843 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.574841 (+ 1998us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.575077 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.575164 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.575408 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1973,"thread_start_us":1901,"threads_started":1}
I20260619 04:49:04.580598 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.564066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.0.0.1:58572)
0619 04:49:04.569028 (+ 4962us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:04.569051 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:04.569159 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:04.571520 (+ 2361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:04.571528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:04.571563 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:04.572081 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.572096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.574915 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.574923 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.577837 (+ 2914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.577853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.579738 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.579748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.579952 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.579987 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:49:04.580143 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 04:49:04.580269 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4794,"thread_start_us":433,"threads_started":1}
I20260619 04:49:04.580698 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.564568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58572 (local address 127.3.243.129:43111)
0619 04:49:04.568657 (+ 4089us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.568666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.569300 (+ 634us) server_negotiation.cc:407] Connection header received
0619 04:49:04.569424 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.569434 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.570956 (+ 1522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.571195 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:04.572362 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.574703 (+ 2341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.578032 (+ 3329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.579230 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.579359 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.579581 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.580197 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.580382 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3900,"thread_start_us":3719,"threads_started":1}
I20260619 04:49:04.581945 19829 tablet_service.cc:1467] Processing CreateTablet for tablet e894537f26fa4856853694f7b2b06c76 (DEFAULT_TABLE table=test-table [id=40a43071a67143d1b8925c7513717ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:49:04.583505 19829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e894537f26fa4856853694f7b2b06c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:49:04.583637 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.569122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:44585 (local address 127.0.0.1:44110)
0619 04:49:04.574564 (+ 5442us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:04.574582 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:04.574675 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:04.577155 (+ 2480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:04.577160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:04.577182 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:04.577361 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.577371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.579372 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.579377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.581275 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.581293 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.582940 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.582946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.583114 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.583144 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 04:49:04.583249 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 04:49:04.583376 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5360,"thread_start_us":8429,"threads_started":1}
I20260619 04:49:04.583796 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.569624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44110 (local address 127.3.243.131:44585)
0619 04:49:04.576713 (+ 7089us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.576720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.576746 (+ 26us) server_negotiation.cc:407] Connection header received
0619 04:49:04.576818 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.576824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.576985 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.577181 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:04.577570 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.579158 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.581426 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.582753 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.582911 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.583121 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.583307 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.583454 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6940,"thread_start_us":5435,"threads_started":1}
I20260619 04:49:04.584460 19695 tablet_service.cc:1467] Processing CreateTablet for tablet e894537f26fa4856853694f7b2b06c76 (DEFAULT_TABLE table=test-table [id=40a43071a67143d1b8925c7513717ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:49:04.585948 19695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e894537f26fa4856853694f7b2b06c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:49:04.587661 19962 tablet_service.cc:1467] Processing CreateTablet for tablet e894537f26fa4856853694f7b2b06c76 (DEFAULT_TABLE table=test-table [id=40a43071a67143d1b8925c7513717ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:49:04.589103 19962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e894537f26fa4856853694f7b2b06c76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:49:04.597662 20045 tablet_bootstrap.cc:492] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: Bootstrap starting.
I20260619 04:49:04.600853 20046 tablet_bootstrap.cc:492] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: Bootstrap starting.
I20260619 04:49:04.602358 20045 tablet_bootstrap.cc:654] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:49:04.603662 20045 log.cc:824] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: Log is configured to *not* fsync() on all Append() calls
I20260619 04:49:04.604074 20047 tablet_bootstrap.cc:492] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: Bootstrap starting.
I20260619 04:49:04.604501 20046 tablet_bootstrap.cc:654] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: Neither blocks nor log segments found. Creating new log.
I20260619 04:49:04.606048 20046 log.cc:824] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: Log is configured to *not* fsync() on all Append() calls
I20260619 04:49:04.607228 20047 tablet_bootstrap.cc:654] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: Neither blocks nor log segments found. Creating new log.
I20260619 04:49:04.608441 20047 log.cc:824] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 04:49:04.608728 20045 tablet_bootstrap.cc:492] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: No bootstrap required, opened a new log
I20260619 04:49:04.609042 20045 ts_tablet_manager.cc:1397] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260619 04:49:04.609210 20046 tablet_bootstrap.cc:492] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: No bootstrap required, opened a new log
I20260619 04:49:04.609477 20046 ts_tablet_manager.cc:1397] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 04:49:04.611465 20047 tablet_bootstrap.cc:492] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: No bootstrap required, opened a new log
I20260619 04:49:04.611768 20047 ts_tablet_manager.cc:1397] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260619 04:49:04.616796 20045 raft_consensus.cc:348] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } }
I20260619 04:49:04.617161 20045 raft_consensus.cc:374] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:49:04.617257 20045 raft_consensus.cc:729] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6974b0ea51884bed912d50e8209023e9, State: Initialized, Role: FOLLOWER
I20260619 04:49:04.617869 20045 consensus_queue.cc:260] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } }
I20260619 04:49:04.619630 20045 ts_tablet_manager.cc:1428] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260619 04:49:04.619918 20046 raft_consensus.cc:348] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } }
I20260619 04:49:04.620368 20046 raft_consensus.cc:374] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:49:04.620455 20046 raft_consensus.cc:729] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d775e9fd77ca487d9a05960d898ac193, State: Initialized, Role: FOLLOWER
I20260619 04:49:04.621026 20046 consensus_queue.cc:260] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } }
I20260619 04:49:04.622164 19894 heartbeater.cc:503] Master 127.3.243.190:44279 was elected leader, sending a full tablet report...
I20260619 04:49:04.622495 20047 raft_consensus.cc:348] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } }
I20260619 04:49:04.622872 20046 ts_tablet_manager.cc:1428] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260619 04:49:04.622920 20047 raft_consensus.cc:374] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:49:04.623000 20047 raft_consensus.cc:729] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d43d16d595448b98dd92aec65a41f4e, State: Initialized, Role: FOLLOWER
I20260619 04:49:04.623764 20047 consensus_queue.cc:260] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } }
I20260619 04:49:04.624966 20027 heartbeater.cc:503] Master 127.3.243.190:44279 was elected leader, sending a full tablet report...
I20260619 04:49:04.625785 20047 ts_tablet_manager.cc:1428] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260619 04:49:04.634482 20052 raft_consensus.cc:482] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:49:04.634757 20052 raft_consensus.cc:504] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } }
I20260619 04:49:04.636303 20052 leader_election.cc:288] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d43d16d595448b98dd92aec65a41f4e (127.3.243.131:44585), 6974b0ea51884bed912d50e8209023e9 (127.3.243.129:43111)
I20260619 04:49:04.645504 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.636667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:42949 (local address 127.3.243.129:43111)
0619 04:49:04.636865 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.636871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.637896 (+ 1025us) server_negotiation.cc:407] Connection header received
0619 04:49:04.638020 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.638025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.638110 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.638227 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:04.638991 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.640141 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.643128 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.644354 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.644494 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.644565 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.645138 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.645258 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:49:04.645499 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.636735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:50411 (local address 127.3.243.131:44585)
0619 04:49:04.636898 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.636905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.637827 (+ 922us) server_negotiation.cc:407] Connection header received
0619 04:49:04.638020 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.638025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.638146 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.638231 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:04.638987 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.640142 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.643118 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.644338 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.644494 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.644565 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.645138 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.645257 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:49:04.645879 20055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.636713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:44585 (local address 127.3.243.130:50411)
0619 04:49:04.637712 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:04.637732 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:04.637887 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:04.638264 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:04.638268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:04.638302 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:04.638819 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.638838 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.640263 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.640269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.642961 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.642977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.644592 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.644598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.644900 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.645007 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 04:49:04.645102 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 04:49:04.645297 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":357,"thread_start_us":603,"threads_started":1}
I20260619 04:49:04.646157 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.636762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.3.243.130:42949)
0619 04:49:04.637800 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:04.637814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:04.637906 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:04.638264 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:04.638268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:04.638359 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:04.638853 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.638861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.640263 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.640269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.642961 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:04.642977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.644596 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.644600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:04.644898 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.645011 (+ 113us) client_negotiation.cc:769] Sending connection context
0619 04:49:04.645102 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 04:49:04.645297 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":278,"spinlock_wait_cycles":30976,"thread_start_us":514,"threads_started":1}
I20260619 04:49:04.647001 19982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e894537f26fa4856853694f7b2b06c76" candidate_uuid: "d775e9fd77ca487d9a05960d898ac193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d43d16d595448b98dd92aec65a41f4e" is_pre_election: true
I20260619 04:49:04.647055 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e894537f26fa4856853694f7b2b06c76" candidate_uuid: "d775e9fd77ca487d9a05960d898ac193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6974b0ea51884bed912d50e8209023e9" is_pre_election: true
I20260619 04:49:04.647586 19982 raft_consensus.cc:2454] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d775e9fd77ca487d9a05960d898ac193 in term 0.
I20260619 04:49:04.647585 19715 raft_consensus.cc:2454] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d775e9fd77ca487d9a05960d898ac193 in term 0.
I20260619 04:49:04.648319 19784 leader_election.cc:302] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6974b0ea51884bed912d50e8209023e9, 8d43d16d595448b98dd92aec65a41f4e, d775e9fd77ca487d9a05960d898ac193; no voters:
I20260619 04:49:04.648788 20052 raft_consensus.cc:2781] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:49:04.648941 20052 raft_consensus.cc:482] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:49:04.649050 20052 raft_consensus.cc:3037] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:49:04.651181 20052 raft_consensus.cc:504] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } }
I20260619 04:49:04.651794 20052 leader_election.cc:288] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [CANDIDATE]: Term 1 election: Requested vote from peers 8d43d16d595448b98dd92aec65a41f4e (127.3.243.131:44585), 6974b0ea51884bed912d50e8209023e9 (127.3.243.129:43111)
I20260619 04:49:04.652230 19982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e894537f26fa4856853694f7b2b06c76" candidate_uuid: "d775e9fd77ca487d9a05960d898ac193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d43d16d595448b98dd92aec65a41f4e"
I20260619 04:49:04.652305 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e894537f26fa4856853694f7b2b06c76" candidate_uuid: "d775e9fd77ca487d9a05960d898ac193" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6974b0ea51884bed912d50e8209023e9"
I20260619 04:49:04.652427 19982 raft_consensus.cc:3037] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:49:04.652559 19715 raft_consensus.cc:3037] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:49:04.654671 19982 raft_consensus.cc:2454] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d775e9fd77ca487d9a05960d898ac193 in term 1.
I20260619 04:49:04.654683 19715 raft_consensus.cc:2454] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d775e9fd77ca487d9a05960d898ac193 in term 1.
I20260619 04:49:04.655057 19782 leader_election.cc:302] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d43d16d595448b98dd92aec65a41f4e, d775e9fd77ca487d9a05960d898ac193; no voters:
I20260619 04:49:04.655381 20052 raft_consensus.cc:2781] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:49:04.655812 20052 raft_consensus.cc:686] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 1 LEADER]: Becoming Leader. State: Replica: d775e9fd77ca487d9a05960d898ac193, State: Running, Role: LEADER
I20260619 04:49:04.656337 20052 consensus_queue.cc:237] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } }
I20260619 04:49:04.661053 19570 catalog_manager.cc:5495] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 reported cstate change: term changed from 0 to 1, leader changed from <none> to d775e9fd77ca487d9a05960d898ac193 (127.3.243.130). New cstate: current_term: 1 leader_uuid: "d775e9fd77ca487d9a05960d898ac193" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } health_report { overall_health: UNKNOWN } } }
I20260619 04:49:04.698431 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.693285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55422 (local address 127.3.243.130:42135)
0619 04:49:04.693467 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.693471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.693591 (+ 120us) server_negotiation.cc:407] Connection header received
0619 04:49:04.693766 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.693771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.693848 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.693979 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:49:04.694313 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.695010 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.696462 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.696858 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.696950 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.697380 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:49:04.697952 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:49:04.698056 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.698152 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.698215 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:49:04.706770 19715 raft_consensus.cc:1261] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [term 1 FOLLOWER]: Refusing update from remote peer d775e9fd77ca487d9a05960d898ac193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:49:04.706789 19982 raft_consensus.cc:1261] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [term 1 FOLLOWER]: Refusing update from remote peer d775e9fd77ca487d9a05960d898ac193: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:49:04.707741 20056 consensus_queue.cc:1035] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:49:04.708140 20052 consensus_queue.cc:1035] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:49:04.719981 20060 mvcc.cc:205] Tried to move back new op lower bound from 7298435255109181440 to 7298435254915891200. Current Snapshot: MvccSnapshot[applied={T|T < 7298435255109181440}]
I20260619 04:49:04.720463 20059 mvcc.cc:205] Tried to move back new op lower bound from 7298435255109181440 to 7298435254915891200. Current Snapshot: MvccSnapshot[applied={T|T < 7298435255109181440}]
I20260619 04:49:04.723740 20061 mvcc.cc:205] Tried to move back new op lower bound from 7298435255109181440 to 7298435254915891200. Current Snapshot: MvccSnapshot[applied={T|T < 7298435255109181440}]
I20260619 04:49:04.734625 19570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:49:04.734927 19570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:49:04.738804 19570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Sending DeleteTablet for 3 replicas of tablet e894537f26fa4856853694f7b2b06c76
I20260619 04:49:04.739923 19829 tablet_service.cc:1514] Processing DeleteTablet for tablet e894537f26fa4856853694f7b2b06c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:49:04 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:55414
I20260619 04:49:04.740031 19962 tablet_service.cc:1514] Processing DeleteTablet for tablet e894537f26fa4856853694f7b2b06c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:49:04 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:44110
I20260619 04:49:04.740450 19695 tablet_service.cc:1514] Processing DeleteTablet for tablet e894537f26fa4856853694f7b2b06c76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:49:04 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:58572
I20260619 04:49:04.740667 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 19632
I20260619 04:49:04.752396 20070 tablet_replica.cc:330] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: stopping tablet replica
I20260619 04:49:04.752902 20070 raft_consensus.cc:2229] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:49:04.753466 20070 raft_consensus.cc:2258] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:49:04.754570 20069 tablet_replica.cc:330] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: stopping tablet replica
I20260619 04:49:04.754938 20071 tablet_replica.cc:330] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: stopping tablet replica
I20260619 04:49:04.755030 20069 raft_consensus.cc:2229] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:49:04.755333 20071 raft_consensus.cc:2229] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:49:04.755719 20069 raft_consensus.cc:2258] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:49:04.755782 20071 raft_consensus.cc:2258] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:49:04.756063 20070 ts_tablet_manager.cc:1905] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:49:04.757936 20069 ts_tablet_manager.cc:1905] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:49:04.760861 20071 ts_tablet_manager.cc:1905] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:49:04.763073 20070 ts_tablet_manager.cc:1918] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:49:04.763247 20070 log.cc:1196] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/e894537f26fa4856853694f7b2b06c76
I20260619 04:49:04.763820 20070 ts_tablet_manager.cc:1939] T e894537f26fa4856853694f7b2b06c76 P 8d43d16d595448b98dd92aec65a41f4e: Deleting consensus metadata
I20260619 04:49:04.765151 20069 ts_tablet_manager.cc:1918] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:49:04.765311 20069 log.cc:1196] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/e894537f26fa4856853694f7b2b06c76
I20260619 04:49:04.765762 19555 catalog_manager.cc:4841] TS 8d43d16d595448b98dd92aec65a41f4e (127.3.243.131:44585): tablet e894537f26fa4856853694f7b2b06c76 (table test-table [id=40a43071a67143d1b8925c7513717ae6]) successfully deleted
I20260619 04:49:04.765867 20069 ts_tablet_manager.cc:1939] T e894537f26fa4856853694f7b2b06c76 P d775e9fd77ca487d9a05960d898ac193: Deleting consensus metadata
I20260619 04:49:04.767434 19555 catalog_manager.cc:4841] TS d775e9fd77ca487d9a05960d898ac193 (127.3.243.130:42135): tablet e894537f26fa4856853694f7b2b06c76 (table test-table [id=40a43071a67143d1b8925c7513717ae6]) successfully deleted
I20260619 04:49:04.767748 20071 ts_tablet_manager.cc:1918] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:49:04.767936 20071 log.cc:1196] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/e894537f26fa4856853694f7b2b06c76
I20260619 04:49:04.768496 20071 ts_tablet_manager.cc:1939] T e894537f26fa4856853694f7b2b06c76 P 6974b0ea51884bed912d50e8209023e9: Deleting consensus metadata
I20260619 04:49:04.769887 19557 catalog_manager.cc:4841] TS 6974b0ea51884bed912d50e8209023e9 (127.3.243.129:43111): tablet e894537f26fa4856853694f7b2b06c76 (table test-table [id=40a43071a67143d1b8925c7513717ae6]) successfully deleted
Jun 19 04:49:04 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:49:04.779451 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.741938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.3.243.129:43111)
0619 04:49:04.742110 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.742117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.750869 (+ 8752us) server_negotiation.cc:407] Connection header received
0619 04:49:04.752028 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.752036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.752159 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.752238 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:04.756071 (+ 3833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.757201 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.766293 (+ 9092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.766993 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.767113 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.772785 (+ 5672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:04.772801 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:04.772821 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:04.772873 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:04.776492 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:04.777252 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:04.777262 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:04.777266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:04.777430 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:04.777796 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:04.777802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:04.777805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:04.778269 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:04.778563 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.778808 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.779061 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:49:04.780344 19755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:04.871541 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 19632
I20260619 04:49:04.895160 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 19766
Jun 19 04:49:04 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:49:04.913025 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:04.896226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.3.243.130:42135)
0619 04:49:04.896435 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:04.896441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:04.896460 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:49:04.896529 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:04.896533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:04.896593 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:04.896676 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:04.898393 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.899210 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.901122 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:04.901570 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:04.901663 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:04.906870 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:04.906882 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:04.906901 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:04.906947 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:04.910178 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:04.910963 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:04.910970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:04.910972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:04.911109 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:04.911546 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:04.911549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:04.911551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:04.911891 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:04.912230 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:04.912529 (+ 299us) server_negotiation.cc:299] Negotiation successful
0619 04:49:04.912697 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:49:04.913980 19889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:05.027338 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 19766
I20260619 04:49:05.046933 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 19899
Jun 19 04:49:05 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:49:05.062284 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:05.047658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44122 (local address 127.3.243.131:44585)
0619 04:49:05.047853 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:05.047860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:05.047914 (+ 54us) server_negotiation.cc:407] Connection header received
0619 04:49:05.048160 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:05.048165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:05.048252 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:05.048359 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:05.049680 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:05.050445 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:05.051752 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:05.052369 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:05.052495 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:05.056865 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:05.056880 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:05.056895 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:05.056931 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:05.059546 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:05.060305 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:05.060313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:05.060315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:05.060450 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:05.060813 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:05.060816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:05.060818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:05.061169 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:05.061395 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:05.061760 (+ 365us) server_negotiation.cc:299] Negotiation successful
0619 04:49:05.061948 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 04:49:05.062963 20022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:05.152570 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 19899
I20260619 04:49:05.169135 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 19539
I20260619 04:49:05.170339 19601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:05.280638 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 19539
I20260619 04:49:25.301697 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44279
--webserver_interface=127.3.243.190
--webserver_port=35409
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:49:25.634509 20077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:25.634858 20077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:25.635047 20077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:25.642429 20077 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 04:49:25.642532 20077 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 04:49:25.642629 20077 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 04:49:25.642719 20077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:49:25.642810 20077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:25.642932 20077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:25.643013 20077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:49:25.643095 20077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:49:25.653147 20077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=35409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:25.654548 20077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:25.655916 20077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:25.678664 20086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:25.678686 20084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:25.678725 20083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:25.679376 20077 server_base.cc:1034] running on GCE node
I20260619 04:49:25.680344 20077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:25.681998 20077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:25.683295 20077 hybrid_clock.cc:648] HybridClock initialized: now 1781844565683215 us; error 64 us; skew 500 ppm
I20260619 04:49:25.683699 20077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:25 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:25.690558 20077 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:49:25.692437 20077 webserver.cc:460] Webserver started at http://127.3.243.190:35409/ using document root <none> and password file <none>
I20260619 04:49:25.693272 20077 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:25.693429 20077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:25.699615 20077 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 04:49:25.702951 20093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:25.704645 20077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:49:25.704869 20077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "5a56f0f1b0a14868b6769224b8c56255"
format_stamp: "Formatted at 2026-06-19 04:49:02 on dist-test-slave-2qp8"
server_key: "b328a5d2c1d6519708d5183d49b6d6d9"
server_key_iv: "c961c9a0d2e067807a262e4797f9b4b7"
server_key_version: "encryptionkey@0"
I20260619 04:49:25.706063 20077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:25.738219 20077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:25.743166 20077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:25.743443 20077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:25.762847 20077 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44279
I20260619 04:49:25.762871 20144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44279 every 8 connection(s)
I20260619 04:49:25.764613 20077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:49:25.766162 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 20077
I20260619 04:49:25.767230 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:43111
--local_ip_for_outbound_sockets=127.3.243.129
--tserver_master_addrs=127.3.243.190:44279
--webserver_port=43141
--webserver_interface=127.3.243.129
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:49:25.775063 20145 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:49:25.779031 20145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Bootstrap starting.
I20260619 04:49:25.799405 20145 log.cc:824] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Log is configured to *not* fsync() on all Append() calls
I20260619 04:49:25.812346 20145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:49:25.812917 20145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Bootstrap complete.
I20260619 04:49:25.822466 20145 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:25.823472 20145 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a56f0f1b0a14868b6769224b8c56255, State: Initialized, Role: FOLLOWER
I20260619 04:49:25.824256 20145 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:25.824506 20145 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:49:25.824623 20145 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:49:25.824787 20145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 1 FOLLOWER]: Advancing to term 2
I20260619 04:49:25.828783 20145 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:25.829319 20145 leader_election.cc:302] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a56f0f1b0a14868b6769224b8c56255; no voters:
I20260619 04:49:25.829960 20145 leader_election.cc:288] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 04:49:25.830101 20149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 04:49:25.831316 20149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 2 LEADER]: Becoming Leader. State: Replica: 5a56f0f1b0a14868b6769224b8c56255, State: Running, Role: LEADER
I20260619 04:49:25.831851 20149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:25.832407 20145 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:49:25.834416 20151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a56f0f1b0a14868b6769224b8c56255. Latest consensus state: current_term: 2 leader_uuid: "5a56f0f1b0a14868b6769224b8c56255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } } }
I20260619 04:49:25.834800 20151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: This master's current role is: LEADER
I20260619 04:49:25.834530 20150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5a56f0f1b0a14868b6769224b8c56255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } } }
I20260619 04:49:25.834950 20150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: This master's current role is: LEADER
I20260619 04:49:25.835992 20155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:49:25.852013 20155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:49:25.853478 20155 catalog_manager.cc:1259] Loaded cluster ID: 2d31ef88f21348308fce532e483568e7
I20260619 04:49:25.853579 20155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:49:25.857731 20155 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:49:25.859244 20155 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Loaded TSK: 0
I20260619 04:49:25.915556 20155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Generated new TSK 1
I20260619 04:49:25.916006 20155 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 04:49:26.147150 20147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:26.147495 20147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:26.147663 20147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:26.155221 20147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:26.155395 20147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:26.155570 20147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:49:26.165529 20147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:43111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=43141
--tserver_master_addrs=127.3.243.190:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:26.166930 20147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:26.168310 20147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:26.186103 20173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:26.188575 20172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:26.189025 20175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:26.189563 20147 server_base.cc:1034] running on GCE node
I20260619 04:49:26.190378 20147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:26.191905 20147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:26.193164 20147 hybrid_clock.cc:648] HybridClock initialized: now 1781844566193101 us; error 89 us; skew 500 ppm
I20260619 04:49:26.193564 20147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:26 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:26.197489 20147 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:49:26.199191 20147 webserver.cc:460] Webserver started at http://127.3.243.129:43141/ using document root <none> and password file <none>
I20260619 04:49:26.199924 20147 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:26.200069 20147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:26.205741 20147 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 04:49:26.208612 20182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:26.210139 20147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 04:49:26.210316 20147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "6974b0ea51884bed912d50e8209023e9"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6a336f8632e1f25ba03394c5a44fbb5e"
server_key_iv: "d984e87562b13962c3a2f5782a72aeb3"
server_key_version: "encryptionkey@0"
I20260619 04:49:26.211380 20147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:26.227486 20147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:26.231938 20147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:26.232283 20147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:26.233414 20147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:49:26.234913 20147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:49:26.235034 20147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:26.235172 20147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:49:26.235267 20147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:26.268869 20147 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:43111
I20260619 04:49:26.268955 20294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:43111 every 8 connection(s)
I20260619 04:49:26.270648 20147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:49:26.275769 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 20147
I20260619 04:49:26.276427 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:42135
--local_ip_for_outbound_sockets=127.3.243.130
--tserver_master_addrs=127.3.243.190:44279
--webserver_port=40479
--webserver_interface=127.3.243.130
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:49:26 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:49:26.297183 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:26.273684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:54825 (local address 127.3.243.190:44279)
0619 04:49:26.274274 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:26.274285 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:26.275003 (+ 718us) server_negotiation.cc:407] Connection header received
0619 04:49:26.277481 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:26.277504 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:26.277980 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:26.278397 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:26.279833 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:26.281950 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:26.283558 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:26.284446 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:26.284668 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:26.289119 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:26.289127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:26.289143 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:26.289186 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:26.292370 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:26.293364 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:26.293371 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:26.293378 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:26.293542 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:26.294066 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:26.294070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:26.294072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:26.294607 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:26.294760 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:26.295776 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0619 04:49:26.296164 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":267,"threads_started":1}
I20260619 04:49:26.297693 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:26.273921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:54825)
0619 04:49:26.274819 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:26.274872 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:26.276885 (+ 2013us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:26.278505 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:26.278513 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:26.279031 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:49:26.279583 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:26.279601 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:26.282148 (+ 2547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:26.282152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:26.283403 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:26.283417 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:26.284692 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:26.284700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:26.284999 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:26.285758 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:49:26.285792 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:49:26.288813 (+ 3021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:49:26.292532 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:26.292544 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:26.292566 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:26.293228 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:26.293676 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:26.293683 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:26.293690 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:26.293946 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:26.294763 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:49:26.294774 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:49:26.295240 (+ 466us) client_negotiation.cc:769] Sending connection context
0619 04:49:26.295591 (+ 351us) client_negotiation.cc:240] Negotiation successful
0619 04:49:26.296258 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":386,"threads_started":1}
I20260619 04:49:26.300674 20295 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44279
I20260619 04:49:26.300995 20295 heartbeater.cc:464] Registering TS with master...
I20260619 04:49:26.301819 20295 heartbeater.cc:511] Master 127.3.243.190:44279 requested a full tablet report, sending...
I20260619 04:49:26.304080 20110 ts_manager.cc:194] Registered new tserver with Master: 6974b0ea51884bed912d50e8209023e9 (127.3.243.129:43111)
I20260619 04:49:26.305603 20110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:54825
W20260619 04:49:26.611469 20299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:26.611815 20299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:26.611989 20299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:26.619597 20299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:26.619750 20299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:26.619935 20299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:49:26.630077 20299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:42135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=40479
--tserver_master_addrs=127.3.243.190:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:26.631449 20299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:26.632849 20299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:26.646199 20307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:26.646287 20304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:26.646256 20305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:26.647024 20299 server_base.cc:1034] running on GCE node
I20260619 04:49:26.647637 20299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:26.648686 20299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:26.649952 20299 hybrid_clock.cc:648] HybridClock initialized: now 1781844566649907 us; error 45 us; skew 500 ppm
I20260619 04:49:26.650275 20299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:26 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:26.653546 20299 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:49:26.654990 20299 webserver.cc:460] Webserver started at http://127.3.243.130:40479/ using document root <none> and password file <none>
I20260619 04:49:26.655691 20299 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:26.655853 20299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:26.661089 20299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:49:26.663702 20314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:26.665071 20299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:49:26.665239 20299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "d775e9fd77ca487d9a05960d898ac193"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6dea764294db77aef248098aa2aa7c57"
server_key_iv: "b5a7c44e804515c5d964fa4c3c85d9a5"
server_key_version: "encryptionkey@0"
I20260619 04:49:26.666205 20299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:26.686504 20299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:26.690455 20299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:26.690733 20299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:26.691715 20299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:49:26.693074 20299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:49:26.693198 20299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:26.693337 20299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:49:26.693434 20299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:26.727828 20299 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:42135
I20260619 04:49:26.727871 20426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:42135 every 8 connection(s)
I20260619 04:49:26.729369 20299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:49:26.733593 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 20299
I20260619 04:49:26.734416 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:44585
--local_ip_for_outbound_sockets=127.3.243.131
--tserver_master_addrs=127.3.243.190:44279
--webserver_port=45225
--webserver_interface=127.3.243.131
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 04:49:26 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:49:26.753763 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:26.731881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:48491 (local address 127.3.243.190:44279)
0619 04:49:26.732071 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:26.732076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:26.733427 (+ 1351us) server_negotiation.cc:407] Connection header received
0619 04:49:26.736416 (+ 2989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:26.736421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:26.736491 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:26.736569 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:26.738672 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:26.739447 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:26.742010 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:26.742761 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:26.742908 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:26.747422 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:26.747427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:26.747431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:26.747465 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:26.750105 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:26.751106 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:26.751110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:26.751112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:26.751338 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:26.751871 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:26.751875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:26.751877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:26.752227 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:26.752429 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:26.753333 (+ 904us) server_negotiation.cc:299] Negotiation successful
0619 04:49:26.753517 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 04:49:26.755450 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:26.732451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:48491)
0619 04:49:26.733242 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:26.733302 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:26.736065 (+ 2763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:26.736866 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:26.736880 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:26.737654 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:49:26.738480 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:26.738506 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:26.739683 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:26.739692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:26.741845 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:26.741858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:26.743206 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:26.743211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:26.743419 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:26.744218 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:49:26.744254 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:49:26.747154 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:49:26.750264 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:26.750276 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:26.750306 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:26.750973 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:26.751453 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:26.751461 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:26.751486 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:26.751743 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:26.752386 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:49:26.752397 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:49:26.752904 (+ 507us) client_negotiation.cc:769] Sending connection context
0619 04:49:26.753226 (+ 322us) client_negotiation.cc:240] Negotiation successful
0619 04:49:26.753842 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":323,"threads_started":1}
I20260619 04:49:26.757055 20427 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44279
I20260619 04:49:26.757412 20427 heartbeater.cc:464] Registering TS with master...
I20260619 04:49:26.758141 20427 heartbeater.cc:511] Master 127.3.243.190:44279 requested a full tablet report, sending...
I20260619 04:49:26.759564 20110 ts_manager.cc:194] Registered new tserver with Master: d775e9fd77ca487d9a05960d898ac193 (127.3.243.130:42135)
I20260619 04:49:26.760462 20110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:48491
W20260619 04:49:27.070555 20430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:27.070843 20430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:27.071054 20430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:27.078186 20430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:27.078293 20430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:27.078481 20430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:49:27.088120 20430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:44585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=45225
--tserver_master_addrs=127.3.243.190:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:27.089447 20430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:27.090771 20430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:27.103873 20438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:27.103962 20435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:27.103979 20436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:27.105315 20430 server_base.cc:1034] running on GCE node
I20260619 04:49:27.105909 20430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:27.106818 20430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:27.108021 20430 hybrid_clock.cc:648] HybridClock initialized: now 1781844567107984 us; error 27 us; skew 500 ppm
I20260619 04:49:27.108480 20430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:27 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844567, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:27.111687 20430 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:49:27.113123 20430 webserver.cc:460] Webserver started at http://127.3.243.131:45225/ using document root <none> and password file <none>
I20260619 04:49:27.113786 20430 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:27.114003 20430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:27.119670 20430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:49:27.122607 20445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:27.124068 20430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:49:27.124329 20430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "8d43d16d595448b98dd92aec65a41f4e"
format_stamp: "Formatted at 2026-06-19 04:49:04 on dist-test-slave-2qp8"
server_key: "bec03d1e5ab74c60eeb2c34c4497d97c"
server_key_iv: "9de03abf772ae98e686dd826b4cb3ac1"
server_key_version: "encryptionkey@0"
I20260619 04:49:27.125389 20430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:27.140656 20430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:27.144763 20430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:27.145047 20430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:27.146031 20430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:49:27.147369 20430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:49:27.147492 20430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:27.147630 20430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:49:27.147727 20430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:27.181413 20430 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:44585
I20260619 04:49:27.181466 20557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:44585 every 8 connection(s)
I20260619 04:49:27.183000 20430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:49:27.192896 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 20430
Jun 19 04:49:27 dist-test-slave-2qp8 krb5kdc[19526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844567, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:49:27.203657 20297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.185671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:40379 (local address 127.3.243.190:44279)
0619 04:49:27.185854 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.185859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.187080 (+ 1221us) server_negotiation.cc:407] Connection header received
0619 04:49:27.188919 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.188923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.188990 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.189080 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:27.190483 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.191203 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.192788 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.193439 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.193563 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.197411 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:27.197415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:27.197419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:27.197450 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:27.200484 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.201271 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.201277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.201280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.201416 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.201869 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.201874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.201878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.202145 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:27.202317 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.203027 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.203250 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:49:27.204730 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.186205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:40379)
0619 04:49:27.186912 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.186949 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:27.188657 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:27.189276 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:27.189284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:27.189783 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:49:27.190336 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.190353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.191341 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.191344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.192646 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.192657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.193675 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.193685 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.193973 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.194797 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:49:27.194826 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:49:27.197186 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:49:27.200651 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:27.200661 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:27.200679 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:27.201138 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:27.201530 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:49:27.201535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:49:27.201539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:49:27.201738 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:49:27.202278 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:49:27.202286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:49:27.202651 (+ 365us) client_negotiation.cc:769] Sending connection context
0619 04:49:27.202929 (+ 278us) client_negotiation.cc:240] Negotiation successful
0619 04:49:27.203483 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":259,"threads_started":1}
I20260619 04:49:27.206228 20558 heartbeater.cc:347] Connected to a master server at 127.3.243.190:44279
I20260619 04:49:27.206550 20558 heartbeater.cc:464] Registering TS with master...
I20260619 04:49:27.207259 20558 heartbeater.cc:511] Master 127.3.243.190:44279 requested a full tablet report, sending...
I20260619 04:49:27.207909 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.193891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59470 (local address 127.3.243.190:44279)
0619 04:49:27.194429 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.194435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.194724 (+ 289us) server_negotiation.cc:407] Connection header received
0619 04:49:27.194927 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.194937 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.195001 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.195110 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:27.196861 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.197610 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.199097 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.199557 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.199720 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.202458 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:27.202463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:27.202466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:27.202495 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:27.205259 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.205926 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.205938 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.205941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.206047 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.206442 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.206451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.206453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.206738 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:27.207005 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.207330 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.207554 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":280,"threads_started":1}
I20260619 04:49:27.208912 20110 ts_manager.cc:194] Registered new tserver with Master: 8d43d16d595448b98dd92aec65a41f4e (127.3.243.131:44585)
I20260619 04:49:27.209846 20110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:40379
I20260619 04:49:27.210767 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:49:27.217289 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.211715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.3.243.190:44279)
0619 04:49:27.211973 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.211979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.212323 (+ 344us) server_negotiation.cc:407] Connection header received
0619 04:49:27.212507 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.212510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.212627 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.212736 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:49:27.213127 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.213796 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.215418 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.215844 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.215940 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.216503 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:49:27.216762 (+ 259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 04:49:27.217040 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59486: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 04:49:27.217414 20561 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59486: token expired
W20260619 04:49:27.217947 20031 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.243.190:44279)
I20260619 04:49:27.230358 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.218336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.3.243.190:44279)
0619 04:49:27.218477 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.218480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.218578 (+ 98us) server_negotiation.cc:407] Connection header received
0619 04:49:27.218687 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.218690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.218739 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.218813 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:27.220041 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.220773 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.222196 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.222636 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.222721 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.225243 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:27.225247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:27.225250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:27.225278 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:27.227582 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.228340 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.228344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.228346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.228483 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.228835 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.228839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.228842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.229164 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:27.229342 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.229866 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.230096 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:49:27.233798 20109 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:49:27.240993 20109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:49:27.275106 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.254280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:44585 (local address 127.0.0.1:60302)
0619 04:49:27.265660 (+ 11380us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.265682 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:27.265912 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:27.266970 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:27.266974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:27.266996 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:27.267362 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.267378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.269437 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.269443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.271883 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.271902 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.274041 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.274050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.274253 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.274301 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 04:49:27.274441 (+ 140us) client_negotiation.cc:240] Negotiation successful
0619 04:49:27.274633 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11255,"mutex_wait_us":3,"thread_start_us":9660,"threads_started":1}
I20260619 04:49:27.275790 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.254284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.3.243.129:43111)
0619 04:49:27.265205 (+ 10921us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.265217 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.265250 (+ 33us) server_negotiation.cc:407] Connection header received
0619 04:49:27.265390 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.265404 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.265757 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.266613 (+ 856us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:27.266881 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.268656 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.272194 (+ 3538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.273809 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.273967 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.274238 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.275215 (+ 977us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.275416 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10746,"thread_start_us":10618,"threads_started":1}
I20260619 04:49:27.275988 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.265147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59790 (local address 127.3.243.130:42135)
0619 04:49:27.265971 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.265980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.266908 (+ 928us) server_negotiation.cc:407] Connection header received
0619 04:49:27.267006 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.267014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.267243 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.267447 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:27.268249 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.270014 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.272637 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.274030 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.274156 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.274340 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.275492 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.275661 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":507,"threads_started":1}
I20260619 04:49:27.276043 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.254984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.3.243.131:44585)
0619 04:49:27.255915 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.255930 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.265798 (+ 9868us) server_negotiation.cc:407] Connection header received
0619 04:49:27.266180 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.266192 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.266464 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.266665 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:27.267581 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.269228 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.272086 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.273532 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.273655 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.273898 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.275578 (+ 1680us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.275719 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":501,"threads_started":1}
I20260619 04:49:27.276511 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.254199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.0.0.1:45792)
0619 04:49:27.264382 (+ 10183us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.264423 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:27.264597 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:27.266014 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:27.266022 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:27.266055 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:27.266678 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.266698 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.268837 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.268843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.271937 (+ 3094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.271952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.274396 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.274402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.274596 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.274631 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:49:27.274731 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 04:49:27.274885 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9919,"mutex_wait_us":80,"thread_start_us":11180,"threads_started":1}
I20260619 04:49:27.276703 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.266000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:42135 (local address 127.0.0.1:59790)
0619 04:49:27.266663 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.266681 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:27.266792 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:27.267671 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:27.267678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:27.267702 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:27.268059 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.268072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.270171 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.270176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.272464 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.272477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.274488 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.274493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.274652 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.274680 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:49:27.274768 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 04:49:27.274895 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":4,"thread_start_us":338,"threads_started":1}
I20260619 04:49:27.280231 20493 tablet_service.cc:1467] Processing CreateTablet for tablet 8064b343a61447aa80dec3b5c81e3e4a (DEFAULT_TABLE table=test-table [id=a424ba407de8473cb58bdb49f257fd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:49:27.280558 20230 tablet_service.cc:1467] Processing CreateTablet for tablet 8064b343a61447aa80dec3b5c81e3e4a (DEFAULT_TABLE table=test-table [id=a424ba407de8473cb58bdb49f257fd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:49:27.281863 20493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8064b343a61447aa80dec3b5c81e3e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:49:27.282063 20230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8064b343a61447aa80dec3b5c81e3e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:49:27.281854 20362 tablet_service.cc:1467] Processing CreateTablet for tablet 8064b343a61447aa80dec3b5c81e3e4a (DEFAULT_TABLE table=test-table [id=a424ba407de8473cb58bdb49f257fd23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:49:27.283087 20362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8064b343a61447aa80dec3b5c81e3e4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:49:27.296182 20574 tablet_bootstrap.cc:492] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Bootstrap starting.
I20260619 04:49:27.299063 20576 tablet_bootstrap.cc:492] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: Bootstrap starting.
I20260619 04:49:27.300081 20574 tablet_bootstrap.cc:654] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Neither blocks nor log segments found. Creating new log.
I20260619 04:49:27.301088 20575 tablet_bootstrap.cc:492] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: Bootstrap starting.
I20260619 04:49:27.301604 20574 log.cc:824] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Log is configured to *not* fsync() on all Append() calls
I20260619 04:49:27.302562 20576 tablet_bootstrap.cc:654] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: Neither blocks nor log segments found. Creating new log.
I20260619 04:49:27.303709 20576 log.cc:824] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: Log is configured to *not* fsync() on all Append() calls
I20260619 04:49:27.304986 20575 tablet_bootstrap.cc:654] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: Neither blocks nor log segments found. Creating new log.
I20260619 04:49:27.305274 20574 tablet_bootstrap.cc:492] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: No bootstrap required, opened a new log
I20260619 04:49:27.305886 20574 ts_tablet_manager.cc:1397] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20260619 04:49:27.306285 20575 log.cc:824] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: Log is configured to *not* fsync() on all Append() calls
I20260619 04:49:27.306931 20576 tablet_bootstrap.cc:492] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: No bootstrap required, opened a new log
I20260619 04:49:27.307195 20576 ts_tablet_manager.cc:1397] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 04:49:27.308653 20295 heartbeater.cc:503] Master 127.3.243.190:44279 was elected leader, sending a full tablet report...
I20260619 04:49:27.309454 20575 tablet_bootstrap.cc:492] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: No bootstrap required, opened a new log
I20260619 04:49:27.309700 20575 ts_tablet_manager.cc:1397] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 04:49:27.316821 20574 raft_consensus.cc:348] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } }
I20260619 04:49:27.317198 20574 raft_consensus.cc:374] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:49:27.317302 20574 raft_consensus.cc:729] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6974b0ea51884bed912d50e8209023e9, State: Initialized, Role: FOLLOWER
I20260619 04:49:27.317945 20574 consensus_queue.cc:260] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } }
I20260619 04:49:27.318420 20576 raft_consensus.cc:348] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } }
I20260619 04:49:27.318930 20576 raft_consensus.cc:374] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:49:27.319059 20576 raft_consensus.cc:729] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d775e9fd77ca487d9a05960d898ac193, State: Initialized, Role: FOLLOWER
I20260619 04:49:27.319832 20576 consensus_queue.cc:260] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } }
I20260619 04:49:27.320924 20574 ts_tablet_manager.cc:1428] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Time spent starting tablet: real 0.015s user 0.007s sys 0.005s
I20260619 04:49:27.321157 20575 raft_consensus.cc:348] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } }
I20260619 04:49:27.321712 20575 raft_consensus.cc:374] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:49:27.321828 20575 raft_consensus.cc:729] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d43d16d595448b98dd92aec65a41f4e, State: Initialized, Role: FOLLOWER
I20260619 04:49:27.322052 20427 heartbeater.cc:503] Master 127.3.243.190:44279 was elected leader, sending a full tablet report...
I20260619 04:49:27.322799 20576 ts_tablet_manager.cc:1428] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260619 04:49:27.322552 20575 consensus_queue.cc:260] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } }
I20260619 04:49:27.323827 20558 heartbeater.cc:503] Master 127.3.243.190:44279 was elected leader, sending a full tablet report...
I20260619 04:49:27.324532 20575 ts_tablet_manager.cc:1428] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260619 04:49:27.418474 20580 raft_consensus.cc:482] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:49:27.418819 20580 raft_consensus.cc:504] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } }
I20260619 04:49:27.420691 20580 leader_election.cc:288] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d43d16d595448b98dd92aec65a41f4e (127.3.243.131:44585), d775e9fd77ca487d9a05960d898ac193 (127.3.243.130:42135)
I20260619 04:49:27.427568 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.421150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:47073 (local address 127.3.243.131:44585)
0619 04:49:27.421336 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.421340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.421872 (+ 532us) server_negotiation.cc:407] Connection header received
0619 04:49:27.422079 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.422085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.422164 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.422271 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:27.423189 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.423946 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.425867 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.426629 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.426825 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.426903 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.427190 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.427313 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:49:27.427663 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.422139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:39695 (local address 127.3.243.130:42135)
0619 04:49:27.422349 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.422355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.422617 (+ 262us) server_negotiation.cc:407] Connection header received
0619 04:49:27.422861 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.422867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.422934 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.423029 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:49:27.423539 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.424321 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.426029 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.426771 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.426934 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.427014 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.427297 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.427420 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:49:27.427860 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.421043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:44585 (local address 127.3.243.129:47073)
0619 04:49:27.421762 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.421782 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:27.421927 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:27.422525 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:27.422530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:27.422555 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:27.423027 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.423037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.424128 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.424134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.425709 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.425721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.426848 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.426854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.427029 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.427053 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 04:49:27.427178 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 04:49:27.427306 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":45,"thread_start_us":354,"threads_started":1}
I20260619 04:49:27.427954 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.421854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:42135 (local address 127.3.243.129:39695)
0619 04:49:27.422507 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.422521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 04:49:27.422661 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:49:27.423056 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:49:27.423061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:49:27.423083 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:49:27.423392 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.423400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.424443 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.424448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.425880 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:49:27.425891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.426986 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.426992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:49:27.427132 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.427154 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 04:49:27.427270 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 04:49:27.427389 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":19,"thread_start_us":380,"threads_started":1}
I20260619 04:49:27.428994 20382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8064b343a61447aa80dec3b5c81e3e4a" candidate_uuid: "6974b0ea51884bed912d50e8209023e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d775e9fd77ca487d9a05960d898ac193" is_pre_election: true
I20260619 04:49:27.428996 20513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8064b343a61447aa80dec3b5c81e3e4a" candidate_uuid: "6974b0ea51884bed912d50e8209023e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d43d16d595448b98dd92aec65a41f4e" is_pre_election: true
I20260619 04:49:27.429505 20513 raft_consensus.cc:2454] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6974b0ea51884bed912d50e8209023e9 in term 0.
I20260619 04:49:27.429512 20382 raft_consensus.cc:2454] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6974b0ea51884bed912d50e8209023e9 in term 0.
I20260619 04:49:27.430179 20183 leader_election.cc:302] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6974b0ea51884bed912d50e8209023e9, 8d43d16d595448b98dd92aec65a41f4e; no voters:
I20260619 04:49:27.430558 20580 raft_consensus.cc:2781] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:49:27.430704 20580 raft_consensus.cc:482] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:49:27.430828 20580 raft_consensus.cc:3037] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:49:27.433270 20580 raft_consensus.cc:504] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } }
I20260619 04:49:27.433900 20580 leader_election.cc:288] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [CANDIDATE]: Term 1 election: Requested vote from peers 8d43d16d595448b98dd92aec65a41f4e (127.3.243.131:44585), d775e9fd77ca487d9a05960d898ac193 (127.3.243.130:42135)
I20260619 04:49:27.434386 20513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8064b343a61447aa80dec3b5c81e3e4a" candidate_uuid: "6974b0ea51884bed912d50e8209023e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d43d16d595448b98dd92aec65a41f4e"
I20260619 04:49:27.434423 20382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8064b343a61447aa80dec3b5c81e3e4a" candidate_uuid: "6974b0ea51884bed912d50e8209023e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d775e9fd77ca487d9a05960d898ac193"
I20260619 04:49:27.434607 20513 raft_consensus.cc:3037] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:49:27.434616 20382 raft_consensus.cc:3037] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [term 0 FOLLOWER]: Advancing to term 1
W20260619 04:49:27.434674 20559 tablet.cc:2369] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:49:27.437165 20513 raft_consensus.cc:2454] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6974b0ea51884bed912d50e8209023e9 in term 1.
I20260619 04:49:27.437196 20382 raft_consensus.cc:2454] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6974b0ea51884bed912d50e8209023e9 in term 1.
I20260619 04:49:27.437572 20183 leader_election.cc:302] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6974b0ea51884bed912d50e8209023e9, 8d43d16d595448b98dd92aec65a41f4e; no voters:
I20260619 04:49:27.437846 20580 raft_consensus.cc:2781] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:49:27.438318 20580 raft_consensus.cc:686] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 1 LEADER]: Becoming Leader. State: Replica: 6974b0ea51884bed912d50e8209023e9, State: Running, Role: LEADER
I20260619 04:49:27.438768 20580 consensus_queue.cc:237] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } }
I20260619 04:49:27.443306 20110 catalog_manager.cc:5495] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6974b0ea51884bed912d50e8209023e9 (127.3.243.129). New cstate: current_term: 1 leader_uuid: "6974b0ea51884bed912d50e8209023e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6974b0ea51884bed912d50e8209023e9" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 43111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 } health_report { overall_health: UNKNOWN } } }
W20260619 04:49:27.481366 20428 tablet.cc:2369] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:49:27.504594 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.499220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.3.243.129:43111)
0619 04:49:27.499411 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.499415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.499505 (+ 90us) server_negotiation.cc:407] Connection header received
0619 04:49:27.499693 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.499696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.499773 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.499900 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:49:27.500255 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.500993 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.502334 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.502715 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.502803 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.503438 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:49:27.503942 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:49:27.504048 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.504319 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.504387 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 04:49:27.513252 20513 raft_consensus.cc:1261] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [term 1 FOLLOWER]: Refusing update from remote peer 6974b0ea51884bed912d50e8209023e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:49:27.513252 20382 raft_consensus.cc:1261] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [term 1 FOLLOWER]: Refusing update from remote peer 6974b0ea51884bed912d50e8209023e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:49:27.514259 20580 consensus_queue.cc:1035] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d43d16d595448b98dd92aec65a41f4e" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:49:27.514614 20585 consensus_queue.cc:1035] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d775e9fd77ca487d9a05960d898ac193" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 42135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 04:49:27.522980 20296 tablet.cc:2369] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:49:27.542982 20109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:49:27.543301 20109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:49:27.546854 20109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Sending DeleteTablet for 3 replicas of tablet 8064b343a61447aa80dec3b5c81e3e4a
I20260619 04:49:27.548072 20493 tablet_service.cc:1514] Processing DeleteTablet for tablet 8064b343a61447aa80dec3b5c81e3e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:49:27 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:60302
I20260619 04:49:27.548295 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 20147
I20260619 04:49:27.548280 20362 tablet_service.cc:1514] Processing DeleteTablet for tablet 8064b343a61447aa80dec3b5c81e3e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:49:27 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:59790
I20260619 04:49:27.548856 20230 tablet_service.cc:1514] Processing DeleteTablet for tablet 8064b343a61447aa80dec3b5c81e3e4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:49:27 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:45792
I20260619 04:49:27.549376 20599 tablet_replica.cc:330] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: stopping tablet replica
I20260619 04:49:27.549787 20599 raft_consensus.cc:2229] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:49:27.550288 20599 raft_consensus.cc:2258] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:49:27.551322 20598 tablet_replica.cc:330] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: stopping tablet replica
I20260619 04:49:27.551738 20598 raft_consensus.cc:2229] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:49:27.552321 20600 tablet_replica.cc:330] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: stopping tablet replica
I20260619 04:49:27.552700 20600 raft_consensus.cc:2229] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:49:27.552743 20599 ts_tablet_manager.cc:1905] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:49:27.553860 20598 raft_consensus.cc:2258] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:49:27.553336 20600 raft_consensus.cc:2258] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:49:27.557536 20600 ts_tablet_manager.cc:1905] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:49:27.560789 20599 ts_tablet_manager.cc:1918] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:49:27.560813 20598 ts_tablet_manager.cc:1905] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:49:27.560949 20599 log.cc:1196] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/8064b343a61447aa80dec3b5c81e3e4a
I20260619 04:49:27.561493 20599 ts_tablet_manager.cc:1939] T 8064b343a61447aa80dec3b5c81e3e4a P d775e9fd77ca487d9a05960d898ac193: Deleting consensus metadata
I20260619 04:49:27.563409 20094 catalog_manager.cc:4841] TS d775e9fd77ca487d9a05960d898ac193 (127.3.243.130:42135): tablet 8064b343a61447aa80dec3b5c81e3e4a (table test-table [id=a424ba407de8473cb58bdb49f257fd23]) successfully deleted
I20260619 04:49:27.566871 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.549630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45804 (local address 127.3.243.129:43111)
0619 04:49:27.549814 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.549821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.549843 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:49:27.549904 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.549910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.549996 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.550099 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:27.551964 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.553321 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.555312 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.555954 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.556107 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.559591 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:27.559604 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:27.559620 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:27.559670 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:27.563776 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.564581 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.564589 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.564593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.564756 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.565108 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.565114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.565117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.565565 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:27.565937 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.566220 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.566476 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:49:27.567911 20290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:27.571568 20598 ts_tablet_manager.cc:1918] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:49:27.571753 20598 log.cc:1196] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/8064b343a61447aa80dec3b5c81e3e4a
I20260619 04:49:27.572445 20598 ts_tablet_manager.cc:1939] T 8064b343a61447aa80dec3b5c81e3e4a P 8d43d16d595448b98dd92aec65a41f4e: Deleting consensus metadata
I20260619 04:49:27.574100 20094 catalog_manager.cc:4841] TS 8d43d16d595448b98dd92aec65a41f4e (127.3.243.131:44585): tablet 8064b343a61447aa80dec3b5c81e3e4a (table test-table [id=a424ba407de8473cb58bdb49f257fd23]) successfully deleted
I20260619 04:49:27.666046 20600 ts_tablet_manager.cc:1918] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:49:27.666352 20600 log.cc:1196] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/8064b343a61447aa80dec3b5c81e3e4a
I20260619 04:49:27.666702 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 20147
I20260619 04:49:27.685521 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 20299
W20260619 04:49:27.685534 20096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.129:43111 (error 108)
I20260619 04:49:27.687274 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.686883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.0.0.1:45816)
0619 04:49:27.687047 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.687143 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 04:49:27.687486 20096 catalog_manager.cc:4584] TS 6974b0ea51884bed912d50e8209023e9 (127.3.243.129:43111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8064b343a61447aa80dec3b5c81e3e4a: Network error: Client connection negotiation failed: client connection to 127.3.243.129:43111: connect: Connection refused (error 111)
I20260619 04:49:27.699260 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.686315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.3.243.130:42135)
0619 04:49:27.686490 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.686497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.686619 (+ 122us) server_negotiation.cc:407] Connection header received
0619 04:49:27.686823 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.686829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.686909 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.687030 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:27.688584 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.689314 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.690626 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.691023 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.691159 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.693734 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:27.693751 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:27.693765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:27.693796 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:27.696480 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.697181 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.697191 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.697196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.697329 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.697705 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.697711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.697715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.698093 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:27.698406 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.698692 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.698914 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:49:27.699918 20422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:27.741281 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.740736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.0.0.1:45830)
0619 04:49:27.740982 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.741106 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 04:49:27.742336 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.741934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.0.0.1:45834)
0619 04:49:27.742102 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.742188 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 04:49:27.792150 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 20299
I20260619 04:49:27.802874 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.802365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.0.0.1:45840)
0619 04:49:27.802596 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.802709 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 04:49:27.803960 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.803598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.0.0.1:45850)
0619 04:49:27.803762 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.803830 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 04:49:27.809969 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 20430
I20260619 04:49:27.823333 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.810752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.3.243.131:44585)
0619 04:49:27.810954 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 04:49:27.810958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:49:27.810976 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:49:27.811072 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:49:27.811076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:49:27.811162 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:49:27.811271 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:49:27.812598 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.813296 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.814623 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:49:27.815012 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:49:27.815099 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:49:27.817771 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:49:27.817785 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:49:27.817808 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:49:27.817844 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:49:27.820651 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.821351 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.821358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.821361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.821514 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:49:27.821949 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:49:27.821952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:49:27.821954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:49:27.822295 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:49:27.822558 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 04:49:27.822728 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 04:49:27.822987 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:49:27.823957 20553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:27.884781 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.884269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.0.0.1:45864)
0619 04:49:27.884470 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.884558 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 04:49:27.885824 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:27.885436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:43111 (local address 127.0.0.1:45872)
0619 04:49:27.885637 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:27.885709 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:43111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 04:49:27.913084 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 20430
I20260619 04:49:27.929774 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 20077
I20260619 04:49:27.930866 20140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:49:28.045876 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 20077
I20260619 04:49:48.067893 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44279
--webserver_interface=127.3.243.190
--webserver_port=35409
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:49:48.396942 20606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:48.397276 20606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:48.397445 20606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:48.404703 20606 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 04:49:48.404842 20606 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 04:49:48.404918 20606 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 04:49:48.405019 20606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:49:48.405118 20606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:48.405241 20606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:48.405328 20606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:49:48.405413 20606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:49:48.415386 20606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=35409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:48.416805 20606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:48.418113 20606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:48.430335 20611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:48.430400 20612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:48.430440 20614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:48.431119 20606 server_base.cc:1034] running on GCE node
I20260619 04:49:48.431959 20606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:48.433435 20606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:48.434681 20606 hybrid_clock.cc:648] HybridClock initialized: now 1781844588434603 us; error 56 us; skew 500 ppm
I20260619 04:49:48.435020 20606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:48 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:48.441813 20606 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:49:48.443380 20606 webserver.cc:460] Webserver started at http://127.3.243.190:35409/ using document root <none> and password file <none>
I20260619 04:49:48.444171 20606 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:48.444306 20606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:48.449788 20606 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 04:49:48.452522 20621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:48.453826 20606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:49:48.454002 20606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "5a56f0f1b0a14868b6769224b8c56255"
format_stamp: "Formatted at 2026-06-19 04:49:02 on dist-test-slave-2qp8"
server_key: "b328a5d2c1d6519708d5183d49b6d6d9"
server_key_iv: "c961c9a0d2e067807a262e4797f9b4b7"
server_key_version: "encryptionkey@0"
I20260619 04:49:48.454948 20606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:48.502858 20606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:48.507022 20606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:48.507321 20606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:48.525801 20606 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44279
I20260619 04:49:48.525847 20672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44279 every 8 connection(s)
I20260619 04:49:48.527298 20606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:49:48.531488 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 20606
I20260619 04:49:48.532521 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:43111
--local_ip_for_outbound_sockets=127.3.243.129
--tserver_master_addrs=127.3.243.190:44279
--webserver_port=43141
--webserver_interface=127.3.243.129
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:49:48.535241 20673 sys_catalog.cc:263] Verifying existing consensus state
I20260619 04:49:48.540338 20673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Bootstrap starting.
I20260619 04:49:48.560930 20673 log.cc:824] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Log is configured to *not* fsync() on all Append() calls
I20260619 04:49:48.576537 20673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 04:49:48.577081 20673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Bootstrap complete.
I20260619 04:49:48.585989 20673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:48.586838 20673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a56f0f1b0a14868b6769224b8c56255, State: Initialized, Role: FOLLOWER
I20260619 04:49:48.587487 20673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:48.587718 20673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:49:48.587831 20673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:49:48.588032 20673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 2 FOLLOWER]: Advancing to term 3
I20260619 04:49:48.591668 20673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:48.592200 20673 leader_election.cc:302] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a56f0f1b0a14868b6769224b8c56255; no voters:
I20260619 04:49:48.592893 20673 leader_election.cc:288] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 04:49:48.593091 20677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 04:49:48.594099 20677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [term 3 LEADER]: Becoming Leader. State: Replica: 5a56f0f1b0a14868b6769224b8c56255, State: Running, Role: LEADER
I20260619 04:49:48.594696 20677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } }
I20260619 04:49:48.595041 20673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:49:48.597350 20678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5a56f0f1b0a14868b6769224b8c56255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } } }
I20260619 04:49:48.597803 20678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: This master's current role is: LEADER
I20260619 04:49:48.597340 20679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a56f0f1b0a14868b6769224b8c56255. Latest consensus state: current_term: 3 leader_uuid: "5a56f0f1b0a14868b6769224b8c56255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a56f0f1b0a14868b6769224b8c56255" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44279 } } }
I20260619 04:49:48.598300 20679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255 [sys.catalog]: This master's current role is: LEADER
I20260619 04:49:48.599066 20683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:49:48.615402 20683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:49:48.616961 20683 catalog_manager.cc:1259] Loaded cluster ID: 2d31ef88f21348308fce532e483568e7
I20260619 04:49:48.617877 20683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:49:48.622622 20683 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:49:48.625624 20683 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Loaded TSK: 0
I20260619 04:49:48.625818 20683 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5a56f0f1b0a14868b6769224b8c56255: Loaded TSK: 1
F20260619 04:49:48.627007 20683 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3ed081e9ad google::LogMessage::Fail() at ??:0
@ 0x7f3ed082188a google::LogMessage::SendToLog() at ??:0
@ 0x7f3ed081e4de google::LogMessage::Flush() at ??:0
@ 0x7f3ed081fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3ed45827d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f3ed457173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f3ee62511c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f3ee6250783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f3ee62dc2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f3ee62dc0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f3ee63265e9 std::function<>::operator()() at ??:0
@ 0x7f3ee6254cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f3ee6252c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3ee62da610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
I20260619 04:49:48.680260 20696 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f3ee62da439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3ed346a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3ed346d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3ed346d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3ed3448eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3ed67226db start_thread at ??:0
@ 0x7f3ecf6b861f clone at ??:0
W20260619 04:49:48.911005 20675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:48.911356 20675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:48.911523 20675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:48.919217 20675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:48.919378 20675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:48.919553 20675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:49:48.929970 20675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:43111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=43141
--tserver_master_addrs=127.3.243.190:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:48.931301 20675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:48.932788 20675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:48.945962 20720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:48.945919 20721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:48.946189 20723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:48.947017 20675 server_base.cc:1034] running on GCE node
I20260619 04:49:48.947553 20675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:48.948666 20675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:48.949869 20675 hybrid_clock.cc:648] HybridClock initialized: now 1781844588949834 us; error 29 us; skew 500 ppm
I20260619 04:49:48.950294 20675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:48 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844588, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:48.953461 20675 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:49:48.954962 20675 webserver.cc:460] Webserver started at http://127.3.243.129:43141/ using document root <none> and password file <none>
I20260619 04:49:48.955658 20675 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:48.955811 20675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:48.961429 20675 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:49:48.964324 20730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:48.965637 20675 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:49:48.965808 20675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "6974b0ea51884bed912d50e8209023e9"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6a336f8632e1f25ba03394c5a44fbb5e"
server_key_iv: "d984e87562b13962c3a2f5782a72aeb3"
server_key_version: "encryptionkey@0"
I20260619 04:49:48.966794 20675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:48.978096 20675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:48.982137 20675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:48.982409 20675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:48.983381 20675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:49:48.985173 20737 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 04:49:48.990315 20737 ts_tablet_manager.cc:1852] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 04:49:48.991317 20737 ts_tablet_manager.cc:1905] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:49:48.998404 20737 ts_tablet_manager.cc:1918] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:49:48.998718 20737 ts_tablet_manager.cc:1939] T 8064b343a61447aa80dec3b5c81e3e4a P 6974b0ea51884bed912d50e8209023e9: Deleting consensus metadata
I20260619 04:49:48.999317 20675 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 04:49:48.999434 20675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260619 04:49:48.999537 20675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:49:48.999594 20675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:49.034304 20675 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:43111
I20260619 04:49:49.034358 20843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:43111 every 8 connection(s)
I20260619 04:49:49.035696 20675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:49:49.039853 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 20675
I20260619 04:49:49.040606 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:42135
--local_ip_for_outbound_sockets=127.3.243.130
--tserver_master_addrs=127.3.243.190:44279
--webserver_port=40479
--webserver_interface=127.3.243.130
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:49:49.041198 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:49.038755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:52545)
0619 04:49:49.039693 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:49.039972 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":300,"threads_started":1}
W20260619 04:49:49.041908 20844 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:44279 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.190:44279: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
I20260619 04:49:49.042892 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:49.042455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:48919)
0619 04:49:49.042646 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:49.042745 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 04:49:49.043864 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:49.043438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:55475)
0619 04:49:49.043606 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:49.043706 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 04:49:49.044142 20844 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 04:49:49.369055 20847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:49.369336 20847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:49.369518 20847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:49.376994 20847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:49.377161 20847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:49.377343 20847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:49:49.387022 20847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:42135
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=40479
--tserver_master_addrs=127.3.243.190:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:49.388471 20847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:49.389802 20847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:49.402920 20853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:49.402999 20855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:49.402961 20852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:49.404012 20847 server_base.cc:1034] running on GCE node
I20260619 04:49:49.404706 20847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:49.405630 20847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:49.406848 20847 hybrid_clock.cc:648] HybridClock initialized: now 1781844589406807 us; error 36 us; skew 500 ppm
I20260619 04:49:49.407207 20847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:49 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844589, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:49.410578 20847 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:49:49.412250 20847 webserver.cc:460] Webserver started at http://127.3.243.130:40479/ using document root <none> and password file <none>
I20260619 04:49:49.412957 20847 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:49.413093 20847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:49.418550 20847 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 04:49:49.421284 20862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:49.422662 20847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:49:49.422880 20847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "d775e9fd77ca487d9a05960d898ac193"
format_stamp: "Formatted at 2026-06-19 04:49:03 on dist-test-slave-2qp8"
server_key: "6dea764294db77aef248098aa2aa7c57"
server_key_iv: "b5a7c44e804515c5d964fa4c3c85d9a5"
server_key_version: "encryptionkey@0"
I20260619 04:49:49.423853 20847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:49.466723 20847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:49.471037 20847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:49.471359 20847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:49.472451 20847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:49:49.473790 20847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:49:49.473918 20847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:49.474054 20847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:49:49.474143 20847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:49.506518 20847 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:42135
I20260619 04:49:49.506572 20974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:42135 every 8 connection(s)
I20260619 04:49:49.507965 20847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:49:49.512784 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:49.510940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:42785)
0619 04:49:49.511645 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:49.511884 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":252,"threads_started":1}
W20260619 04:49:49.513419 20975 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:44279 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.190:44279: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
I20260619 04:49:49.514209 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:49.513893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:45499)
0619 04:49:49.514037 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:49.514109 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 04:49:49.514984 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:49.514680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:43983)
0619 04:49:49.514812 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:49.514887 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 04:49:49.515232 20975 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 04:49:49.515707 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 20847
I20260619 04:49:49.516332 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:44585
--local_ip_for_outbound_sockets=127.3.243.131
--tserver_master_addrs=127.3.243.190:44279
--webserver_port=45225
--webserver_interface=127.3.243.131
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:39491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:49:49.849624 20978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:49:49.849895 20978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:49:49.850078 20978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:49:49.857157 20978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:49:49.857254 20978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:49:49.857427 20978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:49:49.866861 20978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:39491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:44585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=45225
--tserver_master_addrs=127.3.243.190:44279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:49:49.868098 20978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:49:49.869434 20978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:49:49.882499 20983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:49.882460 20986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:49:49.882526 20984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:49:49.883590 20978 server_base.cc:1034] running on GCE node
I20260619 04:49:49.884222 20978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:49:49.885113 20978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:49:49.886373 20978 hybrid_clock.cc:648] HybridClock initialized: now 1781844589886320 us; error 57 us; skew 500 ppm
I20260619 04:49:49.886801 20978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:49:49 dist-test-slave-2qp8 krb5kdc[19526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844589, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:49:49.890102 20978 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:49:49.891515 20978 webserver.cc:460] Webserver started at http://127.3.243.131:45225/ using document root <none> and password file <none>
I20260619 04:49:49.892261 20978 fs_manager.cc:359] Metadata directory not provided
I20260619 04:49:49.892395 20978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:49:49.897771 20978 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:49:49.900578 20993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:49.901968 20978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:49:49.902173 20978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "8d43d16d595448b98dd92aec65a41f4e"
format_stamp: "Formatted at 2026-06-19 04:49:04 on dist-test-slave-2qp8"
server_key: "bec03d1e5ab74c60eeb2c34c4497d97c"
server_key_iv: "9de03abf772ae98e686dd826b4cb3ac1"
server_key_version: "encryptionkey@0"
I20260619 04:49:49.903185 20978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:49:49.925117 20978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:49:49.929526 20978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:49:49.929822 20978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:49:49.930789 20978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:49:49.932159 20978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:49:49.932278 20978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:49.932408 20978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:49:49.932492 20978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:49:49.965370 20978 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:44585
I20260619 04:49:49.965453 21105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:44585 every 8 connection(s)
I20260619 04:49:49.966867 20978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:49:49.969210 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 20978
I20260619 04:49:49.972436 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:49.970083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:58611)
0619 04:49:49.971181 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:49.971485 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":806,"thread_start_us":586,"threads_started":1}
W20260619 04:49:49.973062 21106 heartbeater.cc:644] Failed to heartbeat to 127.3.243.190:44279 (0 consecutive failures): Network error: Failed to ping master at 127.3.243.190:44279: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
I20260619 04:49:49.973815 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:49.973494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:52209)
0619 04:49:49.973640 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:49.973720 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 04:49:49.974614 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:49.974317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:48827)
0619 04:49:49.974434 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:49.974505 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260619 04:49:49.974876 21106 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 04:49:50.045406 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:50.044645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:37475)
0619 04:49:50.045090 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:50.045194 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":208,"threads_started":1}
I20260619 04:49:50.516724 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:50.515908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:40351)
0619 04:49:50.516386 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:50.516488 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260619 04:49:50.976213 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:50.975380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:35437)
0619 04:49:50.975820 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:50.975909 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":209,"threads_started":1}
I20260619 04:49:51.046967 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:51.046314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:53741)
0619 04:49:51.046718 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:51.046796 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":221,"threads_started":1}
I20260619 04:49:51.518507 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:51.517655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:34813)
0619 04:49:51.518156 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:51.518274 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":241,"threads_started":1}
I20260619 04:49:51.977871 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:51.977171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:44063)
0619 04:49:51.977625 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:51.977711 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260619 04:49:52.049012 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:52.048085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:35793)
0619 04:49:52.048712 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:52.048826 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":321,"threads_started":1}
I20260619 04:49:52.520323 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:52.519563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:60555)
0619 04:49:52.520001 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:52.520082 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":241,"threads_started":1}
I20260619 04:49:52.979552 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:52.978838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:44637)
0619 04:49:52.979300 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:52.979381 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":270,"threads_started":1}
I20260619 04:49:53.050851 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:53.049976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:60597)
0619 04:49:53.050512 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:53.050610 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":258,"threads_started":1}
I20260619 04:49:53.522014 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:53.521243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:39019)
0619 04:49:53.521757 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:53.521846 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":281,"threads_started":1}
I20260619 04:49:53.981191 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:53.980507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:58819)
0619 04:49:53.980936 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:53.981012 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":199,"threads_started":1}
I20260619 04:49:54.052492 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:54.051733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:42131)
0619 04:49:54.052251 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:54.052335 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":235,"threads_started":1}
I20260619 04:49:54.523752 21123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:54.522976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:50381)
0619 04:49:54.523511 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:54.523590 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":309,"threads_started":1}
I20260619 04:49:54.982800 21124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:54.982109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:45577)
0619 04:49:54.982536 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:54.982608 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":219,"threads_started":1}
I20260619 04:49:55.054023 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:55.053354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:35667)
0619 04:49:55.053770 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:55.053849 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":213,"threads_started":1}
I20260619 04:49:55.525266 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:55.524594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:54401)
0619 04:49:55.525010 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:55.525098 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":222,"threads_started":1}
I20260619 04:49:55.984433 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:55.983679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:44089)
0619 04:49:55.984172 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:55.984273 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260619 04:49:56.055627 21128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:56.054891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:53597)
0619 04:49:56.055351 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:56.055428 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":213,"threads_started":1}
I20260619 04:49:56.526907 21129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:56.526095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:35915)
0619 04:49:56.526605 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:56.526683 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":223,"threads_started":1}
I20260619 04:49:56.986179 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:56.985410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:38923)
0619 04:49:56.985888 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:56.985969 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":206,"threads_started":1}
I20260619 04:49:57.057294 21131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:57.056621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:44537)
0619 04:49:57.057049 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:57.057125 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":226,"threads_started":1}
I20260619 04:49:57.528607 21132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:57.527769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:44659)
0619 04:49:57.528354 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:57.528440 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":225,"threads_started":1}
I20260619 04:49:57.987866 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:57.987112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:60731)
0619 04:49:57.987614 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:57.987696 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":292,"threads_started":1}
I20260619 04:49:58.058879 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:58.058141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:53701)
0619 04:49:58.058635 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:58.058729 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":264,"threads_started":1}
I20260619 04:49:58.530234 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:58.529421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:34027)
0619 04:49:58.529947 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:58.530039 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":241,"threads_started":1}
I20260619 04:49:58.989465 21136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:58.988792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:60063)
0619 04:49:58.989231 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:58.989305 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":225,"threads_started":1}
I20260619 04:49:59.060636 21137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:59.059905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:48015)
0619 04:49:59.060395 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:59.060469 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":298,"threads_started":1}
I20260619 04:49:59.531845 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:59.531128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:48133)
0619 04:49:59.531584 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:59.531665 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":270,"threads_started":1}
I20260619 04:49:59.991053 21139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:49:59.990404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:53289)
0619 04:49:59.990820 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 04:49:59.990899 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":226,"threads_started":1}
I20260619 04:50:00.062418 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:00.061705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:55719)
0619 04:50:00.062166 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:00.062256 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":254,"threads_started":1}
I20260619 04:50:00.533440 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:00.532755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:43427)
0619 04:50:00.533170 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:00.533260 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":215,"threads_started":1}
I20260619 04:50:00.992911 21142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:00.992198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:56405)
0619 04:50:00.992637 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:00.992727 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":237,"threads_started":1}
I20260619 04:50:01.064167 21143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:01.063364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:40437)
0619 04:50:01.063843 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:01.063944 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":260,"threads_started":1}
I20260619 04:50:01.535064 21144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:01.534371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:54267)
0619 04:50:01.534811 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:01.534901 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":224,"threads_started":1}
I20260619 04:50:01.994465 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:01.993790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:53347)
0619 04:50:01.994235 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:01.994313 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":244,"threads_started":1}
I20260619 04:50:02.066368 21146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:02.065069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:54467)
0619 04:50:02.066094 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:02.066192 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":920,"thread_start_us":803,"threads_started":1}
I20260619 04:50:02.536713 21147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:02.535924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:59297)
0619 04:50:02.536440 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:02.536535 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":286,"threads_started":1}
I20260619 04:50:02.996376 21148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:02.995514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:54637)
0619 04:50:02.995982 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:02.996083 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":229,"threads_started":1}
I20260619 04:50:03.068192 21149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:03.067371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:51887)
0619 04:50:03.067854 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:03.067948 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":224,"threads_started":1}
I20260619 04:50:03.538532 21150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:03.537764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:48275)
0619 04:50:03.538223 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:03.538353 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
I20260619 04:50:03.997984 21151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:03.997278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:51045)
0619 04:50:03.997708 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:03.997810 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":214,"threads_started":1}
I20260619 04:50:04.069924 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:04.069233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.129:54413)
0619 04:50:04.069650 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:04.069734 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":210,"threads_started":1}
I20260619 04:50:04.540472 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:04.539666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.130:41363)
0619 04:50:04.540087 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:04.540236 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":217,"threads_started":1}
I20260619 04:50:04.976215 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 20675
I20260619 04:50:05.000012 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:04.999252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44279 (local address 127.3.243.131:37963)
0619 04:50:04.999679 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:04.999789 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.190:44279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260619 04:50:05.001338 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:04.977356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.3.243.129:43111)
0619 04:50:04.977981 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:04.978017 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:04.978082 (+ 65us) server_negotiation.cc:407] Connection header received
0619 04:50:04.979874 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:04.979885 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:04.980404 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:04.980694 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:04.981983 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:04.983176 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:04.984488 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:04.985127 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:04.985258 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:04.994721 (+ 9463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:04.994729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:04.994746 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:04.994792 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:04.997831 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:04.998442 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:04.998449 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:04.998456 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:04.998615 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:04.999065 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:04.999070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:04.999074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:04.999830 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:05.000057 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:05.000621 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 04:50:05.001049 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":329,"threads_started":1}
I20260619 04:50:05.002774 20839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:05.088939 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 20675
I20260619 04:50:05.108793 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 20847
I20260619 04:50:05.126760 21158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:05.109809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.3.243.130:42135)
0619 04:50:05.110404 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:05.110417 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:05.110455 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:50:05.112082 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:05.112091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:05.112663 (+ 572us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:05.112960 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:05.114468 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:05.115711 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:05.117179 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:05.117866 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:05.118001 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:05.120668 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:05.120676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:05.120692 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:05.120730 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:05.123747 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:05.124470 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:05.124476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:05.124483 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:05.124643 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:05.124965 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:05.124968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:05.124970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:05.125482 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:05.125704 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:05.126071 (+ 367us) server_negotiation.cc:299] Negotiation successful
0619 04:50:05.126485 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":311,"threads_started":1}
I20260619 04:50:05.128259 20970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:05.212718 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 20847
I20260619 04:50:05.230319 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 20978
I20260619 04:50:05.247665 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:05.231341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.3.243.131:44585)
0619 04:50:05.231943 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:05.231957 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:05.232010 (+ 53us) server_negotiation.cc:407] Connection header received
0619 04:50:05.233675 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:05.233684 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:05.234122 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:05.234430 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:05.235868 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:05.237144 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:05.238395 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:05.239042 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:05.239176 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:05.241769 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:05.241777 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:05.241793 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:05.241834 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:05.244687 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:05.245300 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:05.245306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:05.245312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:05.245490 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:05.245825 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:05.245828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:05.245830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:05.246348 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:05.246565 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:05.246914 (+ 349us) server_negotiation.cc:299] Negotiation successful
0619 04:50:05.247349 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":289,"threads_started":1}
I20260619 04:50:05.249092 21101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:05.331979 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 20978
2026-06-19T04:50:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64299 ms)
[----------] 30 tests from SecurityITest (293085 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:05 dist-test-slave-2qp8 krb5kdc[21164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:05 dist-test-slave-2qp8 krb5kdc[21164](info): set up 2 sockets
Jun 19 04:50:05 dist-test-slave-2qp8 krb5kdc[21164](info): commencing operation
krb5kdc: starting...
W20260619 04:50:06.440475 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:06 dist-test-slave-2qp8 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:50:06.687352 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:43517
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:43517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:07.021329 21177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:07.021606 21177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:07.021780 21177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:07.029037 21177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:07.029152 21177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:07.029259 21177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:07.029335 21177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:07.029399 21177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:07.039327 21177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:43517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:43517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:07.040755 21177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:07.042127 21177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:07.054561 21185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:07.054654 21183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:07.054633 21182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:07.055034 21177 server_base.cc:1034] running on GCE node
I20260619 04:50:07.056011 21177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:07.057471 21177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:07.058733 21177 hybrid_clock.cc:648] HybridClock initialized: now 1781844607058674 us; error 52 us; skew 500 ppm
I20260619 04:50:07.059073 21177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:07 dist-test-slave-2qp8 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:07.062263 21177 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:50:07.063627 21177 webserver.cc:460] Webserver started at http://127.3.243.190:37257/ using document root <none> and password file <none>
I20260619 04:50:07.064393 21177 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:07.064496 21177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:07.064745 21177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:07.067627 21177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "1ebfc739ad94497fbdfc16db99cb1545"
format_stamp: "Formatted at 2026-06-19 04:50:07 on dist-test-slave-2qp8"
server_key: "e5d306ee8bbfb00f816aea536270c9fc"
server_key_iv: "a19fb952a058ce4469ae90b86529a3c2"
server_key_version: "encryptionkey@0"
I20260619 04:50:07.068427 21177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "1ebfc739ad94497fbdfc16db99cb1545"
format_stamp: "Formatted at 2026-06-19 04:50:07 on dist-test-slave-2qp8"
server_key: "e5d306ee8bbfb00f816aea536270c9fc"
server_key_iv: "a19fb952a058ce4469ae90b86529a3c2"
server_key_version: "encryptionkey@0"
I20260619 04:50:07.073756 21177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:50:07.077183 21192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:07.078572 21177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:50:07.078712 21177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "1ebfc739ad94497fbdfc16db99cb1545"
format_stamp: "Formatted at 2026-06-19 04:50:07 on dist-test-slave-2qp8"
server_key: "e5d306ee8bbfb00f816aea536270c9fc"
server_key_iv: "a19fb952a058ce4469ae90b86529a3c2"
server_key_version: "encryptionkey@0"
I20260619 04:50:07.078871 21177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:07.095353 21177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:07.099231 21177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:07.099472 21177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:07.117951 21177 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:43517
I20260619 04:50:07.117975 21243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:43517 every 8 connection(s)
I20260619 04:50:07.119335 21177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:07.121299 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 21177
I20260619 04:50:07.121526 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:07.122200 4046 external_mini_cluster.cc:1428] Setting key cff92cc4a1959a25ab40c079485ae3d6
I20260619 04:50:07.123391 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:50:07 dist-test-slave-2qp8 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:07.137049 21244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545: Bootstrap starting.
I20260619 04:50:07.141095 21244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:07.142297 21244 log.cc:824] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:07.145179 21244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545: No bootstrap required, opened a new log
I20260619 04:50:07.145658 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:07.124038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.3.243.190:43517)
0619 04:50:07.125015 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:07.125030 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:07.125079 (+ 49us) server_negotiation.cc:407] Connection header received
0619 04:50:07.126992 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:07.127018 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:07.127540 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:07.127984 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:07.129229 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:07.131221 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:07.132568 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:07.133573 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:07.133749 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:07.137663 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:07.137673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:07.137690 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:07.137729 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:07.141689 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:07.142365 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:07.142373 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:07.142381 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:07.142585 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:07.142954 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:07.142959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:07.142962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:07.143487 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:07.143792 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:07.144264 (+ 472us) server_negotiation.cc:299] Negotiation successful
0619 04:50:07.144672 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":486,"threads_started":1}
I20260619 04:50:07.152984 21244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfc739ad94497fbdfc16db99cb1545" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43517 } }
I20260619 04:50:07.153343 21244 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:07.153470 21244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ebfc739ad94497fbdfc16db99cb1545, State: Initialized, Role: FOLLOWER
I20260619 04:50:07.154057 21244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfc739ad94497fbdfc16db99cb1545" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43517 } }
I20260619 04:50:07.154287 21244 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:07.154443 21244 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:07.154619 21244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:07.157096 21244 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfc739ad94497fbdfc16db99cb1545" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43517 } }
I20260619 04:50:07.157559 21244 leader_election.cc:302] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ebfc739ad94497fbdfc16db99cb1545; no voters:
I20260619 04:50:07.158175 21244 leader_election.cc:288] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:07.158293 21249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:07.159062 21249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [term 1 LEADER]: Becoming Leader. State: Replica: 1ebfc739ad94497fbdfc16db99cb1545, State: Running, Role: LEADER
I20260619 04:50:07.159847 21249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfc739ad94497fbdfc16db99cb1545" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43517 } }
I20260619 04:50:07.160392 21244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:07.162827 21250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ebfc739ad94497fbdfc16db99cb1545" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfc739ad94497fbdfc16db99cb1545" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43517 } } }
I20260619 04:50:07.162809 21251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ebfc739ad94497fbdfc16db99cb1545. Latest consensus state: current_term: 1 leader_uuid: "1ebfc739ad94497fbdfc16db99cb1545" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebfc739ad94497fbdfc16db99cb1545" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 43517 } } }
I20260619 04:50:07.163241 21250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:07.163266 21251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:07.164829 21255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:07.169438 21255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:07.177985 21255 catalog_manager.cc:1347] Generated new cluster ID: bd6d7e28733548198d46d6da678021a3
I20260619 04:50:07.178102 21255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:07.205171 21255 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:07.206521 21255 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:07.212949 21255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545: Generated new TSK 0
I20260619 04:50:07.213816 21255 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:50:07.272058 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43517
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:07.610582 21270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:07.610858 21270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:07.611047 21270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:07.618384 21270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:07.618513 21270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:07.618781 21270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:50:07.628598 21270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:07.629944 21270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:07.631319 21270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:07.644392 21275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:07.644372 21278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:07.644586 21276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:07.645316 21270 server_base.cc:1034] running on GCE node
I20260619 04:50:07.645941 21270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:07.646872 21270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:07.648169 21270 hybrid_clock.cc:648] HybridClock initialized: now 1781844607648105 us; error 33 us; skew 500 ppm
I20260619 04:50:07.648486 21270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:07 dist-test-slave-2qp8 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:07.651600 21270 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:50:07.653046 21270 webserver.cc:460] Webserver started at http://127.3.243.129:39215/ using document root <none> and password file <none>
I20260619 04:50:07.653724 21270 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:07.653877 21270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:07.654183 21270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:07.657168 21270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "0c986e5a0bf448b0871fd19731da20ad"
format_stamp: "Formatted at 2026-06-19 04:50:07 on dist-test-slave-2qp8"
server_key: "c878ea168b95cb2795b3e3b5a889eb31"
server_key_iv: "eb1e3835c769197e996668327f232553"
server_key_version: "encryptionkey@0"
I20260619 04:50:07.657949 21270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "0c986e5a0bf448b0871fd19731da20ad"
format_stamp: "Formatted at 2026-06-19 04:50:07 on dist-test-slave-2qp8"
server_key: "c878ea168b95cb2795b3e3b5a889eb31"
server_key_iv: "eb1e3835c769197e996668327f232553"
server_key_version: "encryptionkey@0"
I20260619 04:50:07.663254 21270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:50:07.666594 21285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:07.668013 21270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:50:07.668217 21270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "0c986e5a0bf448b0871fd19731da20ad"
format_stamp: "Formatted at 2026-06-19 04:50:07 on dist-test-slave-2qp8"
server_key: "c878ea168b95cb2795b3e3b5a889eb31"
server_key_iv: "eb1e3835c769197e996668327f232553"
server_key_version: "encryptionkey@0"
I20260619 04:50:07.668443 21270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:07.690508 21270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:07.694550 21270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:07.694854 21270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:07.695838 21270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:07.697197 21270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:07.697307 21270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:07.697456 21270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:07.697528 21270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:07.730798 21270 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:37919
I20260619 04:50:07.730845 21397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:37919 every 8 connection(s)
I20260619 04:50:07.732254 21270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:50:07.740046 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 21270
I20260619 04:50:07.740294 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:50:07.740916 4046 external_mini_cluster.cc:1428] Setting key e252c03ca1bfe10dbf99c99f82a3c11b
Jun 19 04:50:07 dist-test-slave-2qp8 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844607, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:07.753489 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:07.734745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:34607 (local address 127.3.243.190:43517)
0619 04:50:07.735217 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:07.735224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:07.736234 (+ 1010us) server_negotiation.cc:407] Connection header received
0619 04:50:07.738034 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:07.738039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:07.738121 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:07.738257 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:07.740073 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:07.740953 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:07.742936 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:07.743376 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:07.743475 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:07.747665 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:07.747671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:07.747681 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:07.747723 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:07.750355 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:07.751111 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:07.751118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:07.751121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:07.751252 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:07.751715 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:07.751720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:07.751724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:07.751988 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:07.752168 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:07.752893 (+ 725us) server_negotiation.cc:299] Negotiation successful
0619 04:50:07.753115 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":214,"threads_started":1}
I20260619 04:50:07.754528 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:07.735333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:43517 (local address 127.3.243.129:34607)
0619 04:50:07.736019 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:07.736057 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:07.737762 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:07.738513 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:07.738522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:07.739074 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:07.739867 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:07.739897 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:07.741144 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:07.741150 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:07.742786 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:07.742802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:07.743609 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:07.743616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:07.743960 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:07.744733 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:50:07.744768 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:07.747449 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:07.750506 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:07.750515 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:07.750534 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:07.750983 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:07.751369 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:07.751374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:07.751378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:07.751594 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:07.752140 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:07.752148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:07.752507 (+ 359us) client_negotiation.cc:769] Sending connection context
0619 04:50:07.752786 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 04:50:07.753352 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":267,"threads_started":1}
I20260619 04:50:07.756271 21398 heartbeater.cc:347] Connected to a master server at 127.3.243.190:43517
I20260619 04:50:07.756582 21398 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:07.757295 21398 heartbeater.cc:511] Master 127.3.243.190:43517 requested a full tablet report, sending...
I20260619 04:50:07.759601 21209 ts_manager.cc:194] Registered new tserver with Master: 0c986e5a0bf448b0871fd19731da20ad (127.3.243.129:37919)
I20260619 04:50:07.761802 21209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:34607
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:50:07.800580 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43517
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:08.122470 21404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:08.122745 21404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:08.122911 21404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:08.130389 21404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:08.130486 21404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:08.130682 21404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:50:08.140194 21404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:08.141417 21404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:08.142712 21404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:08.155197 21409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:08.155172 21410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:08.155145 21412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:08.155937 21404 server_base.cc:1034] running on GCE node
I20260619 04:50:08.156529 21404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:08.157373 21404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:08.158560 21404 hybrid_clock.cc:648] HybridClock initialized: now 1781844608158519 us; error 30 us; skew 500 ppm
I20260619 04:50:08.158864 21404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:08 dist-test-slave-2qp8 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:08.162029 21404 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:50:08.163489 21404 webserver.cc:460] Webserver started at http://127.3.243.130:35279/ using document root <none> and password file <none>
I20260619 04:50:08.164251 21404 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:08.164404 21404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:08.164716 21404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:08.167587 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "287b3e4f0d3a4959982e2df63165bdb8"
format_stamp: "Formatted at 2026-06-19 04:50:08 on dist-test-slave-2qp8"
server_key: "4bffcccc3ebd7494b7b01ae650c06b6f"
server_key_iv: "edb69c1f969f844c5f83f58e5afdc278"
server_key_version: "encryptionkey@0"
I20260619 04:50:08.168474 21404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "287b3e4f0d3a4959982e2df63165bdb8"
format_stamp: "Formatted at 2026-06-19 04:50:08 on dist-test-slave-2qp8"
server_key: "4bffcccc3ebd7494b7b01ae650c06b6f"
server_key_iv: "edb69c1f969f844c5f83f58e5afdc278"
server_key_version: "encryptionkey@0"
I20260619 04:50:08.173707 21404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:50:08.177084 21419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:08.178308 21404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:50:08.178491 21404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "287b3e4f0d3a4959982e2df63165bdb8"
format_stamp: "Formatted at 2026-06-19 04:50:08 on dist-test-slave-2qp8"
server_key: "4bffcccc3ebd7494b7b01ae650c06b6f"
server_key_iv: "edb69c1f969f844c5f83f58e5afdc278"
server_key_version: "encryptionkey@0"
I20260619 04:50:08.178680 21404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:08.198864 21404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:08.204151 21404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:08.204502 21404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:08.205667 21404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:08.207350 21404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:08.207422 21404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:08.207576 21404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:08.207633 21404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:08.240350 21404 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:46773
I20260619 04:50:08.240440 21531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:46773 every 8 connection(s)
I20260619 04:50:08.241698 21404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:50:08.244432 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 21404
I20260619 04:50:08.244621 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:50:08.245090 4046 external_mini_cluster.cc:1428] Setting key 61d5e6e614975ebe9d9a30cc7aea4145
Jun 19 04:50:08 dist-test-slave-2qp8 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:08.265782 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.244234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:52141 (local address 127.3.243.190:43517)
0619 04:50:08.244449 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:08.244455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:08.245787 (+ 1332us) server_negotiation.cc:407] Connection header received
0619 04:50:08.248097 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:08.248101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:08.248225 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:08.248307 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:08.250336 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.251130 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.253589 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.254030 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.254129 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.258511 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:08.258519 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:08.258527 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:08.258570 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:08.262394 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:08.263109 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:08.263118 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:08.263124 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:08.263280 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:08.263677 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:08.263684 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:08.263690 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:08.264077 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:08.264250 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:08.265013 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 04:50:08.265279 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:50:08.266417 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.244768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:43517 (local address 127.3.243.130:52141)
0619 04:50:08.245574 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:08.245634 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:08.247768 (+ 2134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:08.248581 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:08.248596 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:08.249393 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:08.250172 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.250199 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.251280 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.251288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.253450 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.253468 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.254251 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.254260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.254594 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.255427 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:50:08.255462 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:08.258274 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:08.262538 (+ 4264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:08.262544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:08.262556 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:08.262974 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:08.263391 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:08.263394 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:08.263396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:08.263564 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:08.264327 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:08.264333 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:08.264631 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 04:50:08.264859 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 04:50:08.265283 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":296,"threads_started":1}
I20260619 04:50:08.267933 21532 heartbeater.cc:347] Connected to a master server at 127.3.243.190:43517
I20260619 04:50:08.268321 21532 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:08.269130 21532 heartbeater.cc:511] Master 127.3.243.190:43517 requested a full tablet report, sending...
I20260619 04:50:08.270663 21209 ts_manager.cc:194] Registered new tserver with Master: 287b3e4f0d3a4959982e2df63165bdb8 (127.3.243.130:46773)
I20260619 04:50:08.271576 21209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:52141
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:50:08.304999 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43517
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:08.631157 21537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:08.631487 21537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:08.631662 21537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:08.638785 21537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:08.638918 21537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:08.639096 21537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:50:08.648488 21537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:43517
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:08.649752 21537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:08.651037 21537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:08.663800 21542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:08.663853 21545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:08.663887 21543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:08.664717 21537 server_base.cc:1034] running on GCE node
I20260619 04:50:08.665221 21537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:08.666069 21537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:08.667255 21537 hybrid_clock.cc:648] HybridClock initialized: now 1781844608667208 us; error 61 us; skew 500 ppm
I20260619 04:50:08.667574 21537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:08 dist-test-slave-2qp8 krb5kdc[21164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:08.670892 21537 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:50:08.672466 21537 webserver.cc:460] Webserver started at http://127.3.243.131:42751/ using document root <none> and password file <none>
I20260619 04:50:08.673139 21537 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:08.673273 21537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:08.673545 21537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:08.676275 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "94ee35abb7414d019caa912fd579932a"
format_stamp: "Formatted at 2026-06-19 04:50:08 on dist-test-slave-2qp8"
server_key: "9218ff52386bb6e701a43c89aa80b5e7"
server_key_iv: "acaeb468a1bc1497f0b7eeb7ba463b61"
server_key_version: "encryptionkey@0"
I20260619 04:50:08.676982 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "94ee35abb7414d019caa912fd579932a"
format_stamp: "Formatted at 2026-06-19 04:50:08 on dist-test-slave-2qp8"
server_key: "9218ff52386bb6e701a43c89aa80b5e7"
server_key_iv: "acaeb468a1bc1497f0b7eeb7ba463b61"
server_key_version: "encryptionkey@0"
I20260619 04:50:08.681627 21537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 04:50:08.685149 21552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:08.686404 21537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:50:08.686573 21537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "94ee35abb7414d019caa912fd579932a"
format_stamp: "Formatted at 2026-06-19 04:50:08 on dist-test-slave-2qp8"
server_key: "9218ff52386bb6e701a43c89aa80b5e7"
server_key_iv: "acaeb468a1bc1497f0b7eeb7ba463b61"
server_key_version: "encryptionkey@0"
I20260619 04:50:08.686765 21537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:08.703229 21537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:08.707558 21537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:08.707851 21537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:08.708837 21537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:08.710079 21537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:08.710203 21537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:08.710361 21537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:08.710435 21537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:08.744367 21537 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:38465
I20260619 04:50:08.744410 21664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:38465 every 8 connection(s)
I20260619 04:50:08.745631 21537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:50:08.747922 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 21537
I20260619 04:50:08.748145 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:50:08.748667 4046 external_mini_cluster.cc:1428] Setting key b832d57812419ccd2b8e16a380aa9fcd
Jun 19 04:50:08 dist-test-slave-2qp8 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844608, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:08.764948 21398 heartbeater.cc:503] Master 127.3.243.190:43517 was elected leader, sending a full tablet report...
I20260619 04:50:08.765384 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.748235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:50337 (local address 127.3.243.190:43517)
0619 04:50:08.748451 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:08.748457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:08.749687 (+ 1230us) server_negotiation.cc:407] Connection header received
0619 04:50:08.751774 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:08.751778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:08.751842 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:08.751919 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:08.753421 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.754156 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.755591 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.755985 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.756072 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.759230 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:08.759235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:08.759238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:08.759274 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:08.762112 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:08.762942 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:08.762946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:08.762948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:08.763107 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:08.763580 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:08.763586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:08.763590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:08.763993 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:08.764160 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:08.764851 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 04:50:08.765060 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:50:08.766346 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.748685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:43517 (local address 127.3.243.131:50337)
0619 04:50:08.749522 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:08.749572 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:08.751505 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:08.752197 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:08.752204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:08.752694 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:08.753253 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.753270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.754284 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.754288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.755471 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.755482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.756221 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.756227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.756414 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.757007 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:50:08.757034 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:08.759054 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:08.762243 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:08.762250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:08.762280 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:08.762827 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:08.763223 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:08.763227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:08.763230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:08.763464 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:08.764155 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:08.764165 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:08.764486 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 04:50:08.764703 (+ 217us) client_negotiation.cc:240] Negotiation successful
0619 04:50:08.765101 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":341,"threads_started":1}
I20260619 04:50:08.767705 21665 heartbeater.cc:347] Connected to a master server at 127.3.243.190:43517
I20260619 04:50:08.768059 21665 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:08.768851 21665 heartbeater.cc:511] Master 127.3.243.190:43517 requested a full tablet report, sending...
I20260619 04:50:08.770504 21209 ts_manager.cc:194] Registered new tserver with Master: 94ee35abb7414d019caa912fd579932a (127.3.243.131:38465)
I20260619 04:50:08.771309 21209 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:50337
I20260619 04:50:08.775835 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:50:08.793133 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.780347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.3.243.190:43517)
0619 04:50:08.780557 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:08.780565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:08.780871 (+ 306us) server_negotiation.cc:407] Connection header received
0619 04:50:08.781060 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:08.781065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:08.781136 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:08.781248 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:08.782681 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.783583 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.784971 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.785374 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.785504 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.787810 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:08.787824 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:08.787829 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:08.787865 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:08.790344 (+ 2479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:08.791201 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:08.791208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:08.791211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:08.791337 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:08.791684 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:08.791690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:08.791693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:08.792007 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:08.792281 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:08.792612 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 04:50:08.792827 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:50:08.796845 21209 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:50:08.803586 21209 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:50:08.837633 21679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.815082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.0.0.1:56878)
0619 04:50:08.827559 (+ 12477us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:08.827583 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:08.827787 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:08.829093 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:08.829105 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:08.829145 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:08.829612 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.829627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.831967 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.831975 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.834450 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.834464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.836636 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.836642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.836852 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.836911 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 04:50:08.837047 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 04:50:08.837230 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12326,"thread_start_us":12475,"threads_started":1}
I20260619 04:50:08.837679 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.815546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.3.243.129:37919)
0619 04:50:08.826899 (+ 11353us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:08.826912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:08.827983 (+ 1071us) server_negotiation.cc:407] Connection header received
0619 04:50:08.828155 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:08.828167 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:08.828499 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:08.828748 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:08.829847 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.831757 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.834627 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.836093 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.836256 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.836515 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:08.837127 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 04:50:08.837314 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11088,"thread_start_us":11368,"threads_started":1}
I20260619 04:50:08.838567 21677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.814879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:46773 (local address 127.0.0.1:45472)
0619 04:50:08.815831 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:08.815871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:08.816036 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:08.831006 (+ 14970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:08.831014 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:08.831039 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:08.831460 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.831471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.833931 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.833939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.835882 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.835896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.837767 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.837774 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.837959 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.837991 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 04:50:08.838069 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 04:50:08.838185 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":2,"thread_start_us":830,"threads_started":1}
I20260619 04:50:08.839468 21678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.815071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.3.243.130:46773)
0619 04:50:08.830387 (+ 15316us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:08.830399 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:08.830434 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:50:08.830558 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:08.830569 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:08.830827 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:08.831030 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:08.832315 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.833809 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.836067 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.837494 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.838778 (+ 1284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.838968 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:08.839030 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 04:50:08.839192 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15098,"thread_start_us":16164,"threads_started":1}
I20260619 04:50:08.839910 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.815861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:38465 (local address 127.0.0.1:40104)
0619 04:50:08.832337 (+ 16476us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:08.832357 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:08.832479 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:08.833071 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:08.833075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:08.833102 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:08.833532 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.833543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.835159 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.835165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.836966 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:08.836977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.839173 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.839180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:08.839348 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.839376 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:50:08.839472 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 04:50:08.839608 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16329,"thread_start_us":16112,"threads_started":1}
I20260619 04:50:08.840008 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:08.816649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40104 (local address 127.3.243.131:38465)
0619 04:50:08.832498 (+ 15849us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:08.832506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:08.832537 (+ 31us) server_negotiation.cc:407] Connection header received
0619 04:50:08.832636 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:08.832642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:08.832855 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:08.833034 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:08.833709 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.835019 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.837160 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:08.838672 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:08.838797 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:08.839043 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:08.839528 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 04:50:08.839687 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15645,"thread_start_us":15912,"threads_started":1}
I20260619 04:50:08.842064 21333 tablet_service.cc:1467] Processing CreateTablet for tablet 3e69fcb588774b6fa5a22100b79217cb (DEFAULT_TABLE table=test-table [id=39e4401d284e4da3a4122a3ad31f422c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:08.842914 21467 tablet_service.cc:1467] Processing CreateTablet for tablet 3e69fcb588774b6fa5a22100b79217cb (DEFAULT_TABLE table=test-table [id=39e4401d284e4da3a4122a3ad31f422c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:08.843742 21333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e69fcb588774b6fa5a22100b79217cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:08.844336 21467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e69fcb588774b6fa5a22100b79217cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:08.844787 21600 tablet_service.cc:1467] Processing CreateTablet for tablet 3e69fcb588774b6fa5a22100b79217cb (DEFAULT_TABLE table=test-table [id=39e4401d284e4da3a4122a3ad31f422c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:08.845929 21600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e69fcb588774b6fa5a22100b79217cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:08.856782 21683 tablet_bootstrap.cc:492] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: Bootstrap starting.
I20260619 04:50:08.859711 21685 tablet_bootstrap.cc:492] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad: Bootstrap starting.
I20260619 04:50:08.860525 21684 tablet_bootstrap.cc:492] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: Bootstrap starting.
I20260619 04:50:08.861698 21683 tablet_bootstrap.cc:654] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:08.862985 21683 log.cc:824] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:08.863732 21685 tablet_bootstrap.cc:654] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:08.864014 21684 tablet_bootstrap.cc:654] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:08.864908 21685 log.cc:824] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:08.865188 21684 log.cc:824] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:08.866377 21683 tablet_bootstrap.cc:492] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: No bootstrap required, opened a new log
I20260619 04:50:08.866650 21683 ts_tablet_manager.cc:1397] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260619 04:50:08.868173 21684 tablet_bootstrap.cc:492] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: No bootstrap required, opened a new log
I20260619 04:50:08.868185 21685 tablet_bootstrap.cc:492] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad: No bootstrap required, opened a new log
I20260619 04:50:08.868424 21684 ts_tablet_manager.cc:1397] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260619 04:50:08.868424 21685 ts_tablet_manager.cc:1397] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260619 04:50:08.874858 21683 raft_consensus.cc:348] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } }
I20260619 04:50:08.875231 21683 raft_consensus.cc:374] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:08.875350 21683 raft_consensus.cc:729] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94ee35abb7414d019caa912fd579932a, State: Initialized, Role: FOLLOWER
I20260619 04:50:08.875938 21683 consensus_queue.cc:260] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } }
I20260619 04:50:08.877056 21665 heartbeater.cc:503] Master 127.3.243.190:43517 was elected leader, sending a full tablet report...
I20260619 04:50:08.879019 21683 ts_tablet_manager.cc:1428] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 04:50:08.879153 21685 raft_consensus.cc:348] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } }
I20260619 04:50:08.879155 21684 raft_consensus.cc:348] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } }
I20260619 04:50:08.879573 21685 raft_consensus.cc:374] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:08.879573 21684 raft_consensus.cc:374] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:08.879674 21684 raft_consensus.cc:729] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287b3e4f0d3a4959982e2df63165bdb8, State: Initialized, Role: FOLLOWER
I20260619 04:50:08.879678 21685 raft_consensus.cc:729] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c986e5a0bf448b0871fd19731da20ad, State: Initialized, Role: FOLLOWER
I20260619 04:50:08.880330 21684 consensus_queue.cc:260] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } }
I20260619 04:50:08.880321 21685 consensus_queue.cc:260] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } }
I20260619 04:50:08.881534 21532 heartbeater.cc:503] Master 127.3.243.190:43517 was elected leader, sending a full tablet report...
I20260619 04:50:08.882211 21684 ts_tablet_manager.cc:1428] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260619 04:50:08.882469 21685 ts_tablet_manager.cc:1428] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
W20260619 04:50:08.984579 21399 tablet.cc:2369] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:50:08.993772 21533 tablet.cc:2369] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:50:08.997203 21666 tablet.cc:2369] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:09.289052 21690 raft_consensus.cc:482] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:50:09.289424 21690 raft_consensus.cc:504] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } }
I20260619 04:50:09.291229 21690 leader_election.cc:288] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c986e5a0bf448b0871fd19731da20ad (127.3.243.129:37919), 94ee35abb7414d019caa912fd579932a (127.3.243.131:38465)
I20260619 04:50:09.297878 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.291715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:35731 (local address 127.3.243.131:38465)
0619 04:50:09.291936 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:09.291946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:09.292579 (+ 633us) server_negotiation.cc:407] Connection header received
0619 04:50:09.292775 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:09.292779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:09.292836 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:09.292931 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:09.293694 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.294388 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.296193 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.296943 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.297129 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:09.297206 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:09.297498 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 04:50:09.297622 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 04:50:09.297930 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.291704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:46017 (local address 127.3.243.129:37919)
0619 04:50:09.291906 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:09.291915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:09.292385 (+ 470us) server_negotiation.cc:407] Connection header received
0619 04:50:09.292616 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:09.292620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:09.292680 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:09.292775 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:09.293499 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.294236 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.296068 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.296941 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.297123 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:09.297215 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:09.297540 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 04:50:09.297671 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 04:50:09.298110 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.291480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.3.243.130:46017)
0619 04:50:09.292268 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:09.292284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:09.292469 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:09.292790 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:09.292794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:09.292830 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:09.293287 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:09.293303 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.294393 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.294398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:09.295912 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:09.295924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.297231 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.297239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:09.297388 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:09.297409 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 04:50:09.297515 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 04:50:09.297647 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":26,"thread_start_us":436,"threads_started":1}
I20260619 04:50:09.298169 21693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.291639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:38465 (local address 127.3.243.130:35731)
0619 04:50:09.292475 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:09.292490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:09.292632 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:09.292961 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:09.292965 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:09.293023 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:09.293534 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:09.293549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.294523 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.294528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:09.296010 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:09.296021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.297175 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.297182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:09.297343 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:09.297366 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 04:50:09.297473 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 04:50:09.297603 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":81,"thread_start_us":527,"threads_started":1}
I20260619 04:50:09.299083 21620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e69fcb588774b6fa5a22100b79217cb" candidate_uuid: "287b3e4f0d3a4959982e2df63165bdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ee35abb7414d019caa912fd579932a" is_pre_election: true
I20260619 04:50:09.299139 21353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e69fcb588774b6fa5a22100b79217cb" candidate_uuid: "287b3e4f0d3a4959982e2df63165bdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c986e5a0bf448b0871fd19731da20ad" is_pre_election: true
I20260619 04:50:09.299580 21620 raft_consensus.cc:2454] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287b3e4f0d3a4959982e2df63165bdb8 in term 0.
I20260619 04:50:09.299681 21353 raft_consensus.cc:2454] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287b3e4f0d3a4959982e2df63165bdb8 in term 0.
I20260619 04:50:09.300207 21422 leader_election.cc:302] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287b3e4f0d3a4959982e2df63165bdb8, 94ee35abb7414d019caa912fd579932a; no voters:
I20260619 04:50:09.300630 21690 raft_consensus.cc:2781] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:50:09.300798 21690 raft_consensus.cc:482] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:50:09.300918 21690 raft_consensus.cc:3037] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:09.303061 21690 raft_consensus.cc:504] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } }
I20260619 04:50:09.303676 21690 leader_election.cc:288] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [CANDIDATE]: Term 1 election: Requested vote from peers 0c986e5a0bf448b0871fd19731da20ad (127.3.243.129:37919), 94ee35abb7414d019caa912fd579932a (127.3.243.131:38465)
I20260619 04:50:09.304083 21353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e69fcb588774b6fa5a22100b79217cb" candidate_uuid: "287b3e4f0d3a4959982e2df63165bdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c986e5a0bf448b0871fd19731da20ad"
I20260619 04:50:09.304234 21620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e69fcb588774b6fa5a22100b79217cb" candidate_uuid: "287b3e4f0d3a4959982e2df63165bdb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94ee35abb7414d019caa912fd579932a"
I20260619 04:50:09.304319 21353 raft_consensus.cc:3037] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:09.304472 21620 raft_consensus.cc:3037] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:09.306502 21353 raft_consensus.cc:2454] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287b3e4f0d3a4959982e2df63165bdb8 in term 1.
I20260619 04:50:09.306610 21620 raft_consensus.cc:2454] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287b3e4f0d3a4959982e2df63165bdb8 in term 1.
I20260619 04:50:09.306879 21420 leader_election.cc:302] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c986e5a0bf448b0871fd19731da20ad, 287b3e4f0d3a4959982e2df63165bdb8; no voters:
I20260619 04:50:09.307199 21690 raft_consensus.cc:2781] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:09.307628 21690 raft_consensus.cc:686] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 1 LEADER]: Becoming Leader. State: Replica: 287b3e4f0d3a4959982e2df63165bdb8, State: Running, Role: LEADER
I20260619 04:50:09.308182 21690 consensus_queue.cc:237] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } }
I20260619 04:50:09.312613 21208 catalog_manager.cc:5495] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 287b3e4f0d3a4959982e2df63165bdb8 (127.3.243.130). New cstate: current_term: 1 leader_uuid: "287b3e4f0d3a4959982e2df63165bdb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "287b3e4f0d3a4959982e2df63165bdb8" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 46773 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 } health_report { overall_health: UNKNOWN } } }
I20260619 04:50:09.396055 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.390449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45488 (local address 127.3.243.130:46773)
0619 04:50:09.390916 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:09.390920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:09.391100 (+ 180us) server_negotiation.cc:407] Connection header received
0619 04:50:09.391294 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:09.391298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:09.391395 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:09.391541 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:50:09.391942 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.392716 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.394121 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.394499 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.394586 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:09.395033 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:50:09.395577 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:50:09.395691 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:09.395784 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 04:50:09.395847 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":226,"threads_started":1}
I20260619 04:50:09.403868 21353 raft_consensus.cc:1261] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [term 1 FOLLOWER]: Refusing update from remote peer 287b3e4f0d3a4959982e2df63165bdb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:09.403875 21620 raft_consensus.cc:1261] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [term 1 FOLLOWER]: Refusing update from remote peer 287b3e4f0d3a4959982e2df63165bdb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:09.404870 21694 consensus_queue.cc:1035] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c986e5a0bf448b0871fd19731da20ad" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:09.405236 21690 consensus_queue.cc:1035] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94ee35abb7414d019caa912fd579932a" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 38465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:09.417150 21705 mvcc.cc:205] Tried to move back new op lower bound from 7298435520108056576 to 7298435519730089984. Current Snapshot: MvccSnapshot[applied={T|T < 7298435520108056576}]
I20260619 04:50:09.417718 21702 mvcc.cc:205] Tried to move back new op lower bound from 7298435520108056576 to 7298435519730089984. Current Snapshot: MvccSnapshot[applied={T|T < 7298435520108056576}]
I20260619 04:50:09.428965 21207 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:09.429232 21207 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:09.432605 21207 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ebfc739ad94497fbdfc16db99cb1545: Sending DeleteTablet for 3 replicas of tablet 3e69fcb588774b6fa5a22100b79217cb
I20260619 04:50:09.433779 21333 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e69fcb588774b6fa5a22100b79217cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:09 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:56878
I20260619 04:50:09.433760 21467 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e69fcb588774b6fa5a22100b79217cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:09 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:45472
I20260619 04:50:09.434091 21600 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e69fcb588774b6fa5a22100b79217cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:09 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:40104
I20260619 04:50:09.435918 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 21270
I20260619 04:50:09.465461 21715 tablet_replica.cc:330] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: stopping tablet replica
I20260619 04:50:09.470049 21715 raft_consensus.cc:2229] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:50:09.471225 21714 tablet_replica.cc:330] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad: stopping tablet replica
I20260619 04:50:09.471429 21715 raft_consensus.cc:2258] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:09.471671 21716 tablet_replica.cc:330] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: stopping tablet replica
I20260619 04:50:09.472476 21716 raft_consensus.cc:2229] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:09.472723 21714 raft_consensus.cc:2229] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:09.473340 21716 raft_consensus.cc:2258] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:09.473737 21715 ts_tablet_manager.cc:1905] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:09.473944 21714 raft_consensus.cc:2258] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:09.476198 21716 ts_tablet_manager.cc:1905] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 04:50:09 dist-test-slave-2qp8 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:50:09.477906 21714 ts_tablet_manager.cc:1905] T 3e69fcb588774b6fa5a22100b79217cb P 0c986e5a0bf448b0871fd19731da20ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:09.481451 21715 ts_tablet_manager.cc:1918] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:09.481678 21715 log.cc:1196] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/3e69fcb588774b6fa5a22100b79217cb
I20260619 04:50:09.482292 21715 ts_tablet_manager.cc:1939] T 3e69fcb588774b6fa5a22100b79217cb P 287b3e4f0d3a4959982e2df63165bdb8: Deleting consensus metadata
I20260619 04:50:09.484493 21196 catalog_manager.cc:4841] TS 287b3e4f0d3a4959982e2df63165bdb8 (127.3.243.130:46773): tablet 3e69fcb588774b6fa5a22100b79217cb (table test-table [id=39e4401d284e4da3a4122a3ad31f422c]) successfully deleted
I20260619 04:50:09.486244 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.437575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.3.243.129:37919)
0619 04:50:09.437794 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:09.437804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:09.461574 (+ 23770us) server_negotiation.cc:407] Connection header received
0619 04:50:09.462247 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:09.462257 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:09.462381 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:09.462477 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:09.465776 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.467409 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.471590 (+ 4181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.472243 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.472351 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:09.478145 (+ 5794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:09.478157 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:09.478172 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:09.478220 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:09.482262 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:09.483239 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:09.483250 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:09.483256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:09.483441 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:09.483953 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:09.483958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:09.483961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:09.484434 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:09.484729 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:09.485371 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 04:50:09.485631 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:50:09.487135 21393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:09.490412 21716 ts_tablet_manager.cc:1918] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:09.490581 21716 log.cc:1196] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/3e69fcb588774b6fa5a22100b79217cb
I20260619 04:50:09.491187 21716 ts_tablet_manager.cc:1939] T 3e69fcb588774b6fa5a22100b79217cb P 94ee35abb7414d019caa912fd579932a: Deleting consensus metadata
I20260619 04:50:09.493112 21195 catalog_manager.cc:4841] TS 94ee35abb7414d019caa912fd579932a (127.3.243.131:38465): tablet 3e69fcb588774b6fa5a22100b79217cb (table test-table [id=39e4401d284e4da3a4122a3ad31f422c]) successfully deleted
I20260619 04:50:09.577629 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 21270
I20260619 04:50:09.595746 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 21404
W20260619 04:50:09.595790 21193 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.243.129:37919 (error 108)
I20260619 04:50:09.598376 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.597470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.0.0.1:56890)
0619 04:50:09.597969 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:09.598133 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:37919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":226,"threads_started":1}
W20260619 04:50:09.598662 21193 catalog_manager.cc:4584] TS 0c986e5a0bf448b0871fd19731da20ad (127.3.243.129:37919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3e69fcb588774b6fa5a22100b79217cb: Network error: Client connection negotiation failed: client connection to 127.3.243.129:37919: connect: Connection refused (error 111)
Jun 19 04:50:09 dist-test-slave-2qp8 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:50:09.612737 21701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.596991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45490 (local address 127.3.243.130:46773)
0619 04:50:09.597181 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:09.597185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:09.597206 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:50:09.597264 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:09.597268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:09.597346 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:09.597468 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:09.599135 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.599965 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.601331 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.601751 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.601918 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:09.606373 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:09.606389 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:09.606404 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:09.606437 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:09.609709 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:09.610488 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:09.610500 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:09.610502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:09.610620 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:09.611074 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:09.611079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:09.611082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:09.611463 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:09.611810 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:09.612102 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 04:50:09.612377 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:50:09.613457 21527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:09.652503 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.651831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.0.0.1:56892)
0619 04:50:09.652221 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:09.652326 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:37919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
I20260619 04:50:09.653679 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.653239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.0.0.1:56908)
0619 04:50:09.653373 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:09.653453 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:37919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 04:50:09.714694 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.714149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.0.0.1:56914)
0619 04:50:09.714405 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:09.714503 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:37919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260619 04:50:09.715843 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.715439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.0.0.1:56930)
0619 04:50:09.715600 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:09.715693 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:37919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 04:50:09.734037 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 21404
I20260619 04:50:09.753129 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 21537
Jun 19 04:50:09 dist-test-slave-2qp8 krb5kdc[21164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:50:09.770242 21682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.754034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40118 (local address 127.3.243.131:38465)
0619 04:50:09.754231 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:09.754238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:09.754258 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:50:09.754317 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:09.754322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:09.754394 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:09.754502 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:09.756001 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.757164 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.758480 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:09.759140 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:09.759265 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:09.764197 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:09.764208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:09.764222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:09.764255 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:09.767074 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:09.767920 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:09.767938 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:09.767941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:09.768087 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:09.768532 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:09.768540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:09.768546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:09.768967 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:09.769271 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:09.769561 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 04:50:09.769792 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:50:09.771029 21660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:09.797389 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.796755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.0.0.1:56932)
0619 04:50:09.797126 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:09.797247 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:37919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260619 04:50:09.798585 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:09.798080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.0.0.1:56942)
0619 04:50:09.798319 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:09.798427 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:37919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 04:50:09.881520 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 21537
I20260619 04:50:09.899170 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 21177
I20260619 04:50:09.900431 21239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:10.013656 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:10.013032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37919 (local address 127.0.0.1:56958)
0619 04:50:10.013365 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:10.013479 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.129:37919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260619 04:50:10.013758 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 21177
2026-06-19T04:50:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4685 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T04:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:10Z Disabled control of system clock
I20260619 04:50:10.090373 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:33147
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:50:10.416682 21727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:10.416949 21727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:10.417088 21727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:10.424187 21727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:10.424290 21727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:10.424386 21727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:10.424503 21727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:10.424568 21727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:10.434214 21727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:33147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:33147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:10.435582 21727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:10.437471 21727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:10.449633 21733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:10.449712 21735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:10.449749 21732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:10.450624 21727 server_base.cc:1034] running on GCE node
I20260619 04:50:10.451388 21727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:10.452831 21727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:10.454051 21727 hybrid_clock.cc:648] HybridClock initialized: now 1781844610454011 us; error 29 us; skew 500 ppm
I20260619 04:50:10.454463 21727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:10.462018 21727 webserver.cc:460] Webserver started at http://127.3.243.190:45253/ using document root <none> and password file <none>
I20260619 04:50:10.462751 21727 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:10.462870 21727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:10.463214 21727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:10.466202 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "3c5425920e5b45398cdc77ef6f3ef585"
format_stamp: "Formatted at 2026-06-19 04:50:10 on dist-test-slave-2qp8"
server_key: "f3a56dc3f6041e49ebd5692f8bb2b821"
server_key_iv: "0beb6fd36936db98787b0cc67e4c73e7"
server_key_version: "encryptionkey@0"
I20260619 04:50:10.467007 21727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "3c5425920e5b45398cdc77ef6f3ef585"
format_stamp: "Formatted at 2026-06-19 04:50:10 on dist-test-slave-2qp8"
server_key: "f3a56dc3f6041e49ebd5692f8bb2b821"
server_key_iv: "0beb6fd36936db98787b0cc67e4c73e7"
server_key_version: "encryptionkey@0"
I20260619 04:50:10.472011 21727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260619 04:50:10.475630 21741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:10.476989 21727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:10.477170 21727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "3c5425920e5b45398cdc77ef6f3ef585"
format_stamp: "Formatted at 2026-06-19 04:50:10 on dist-test-slave-2qp8"
server_key: "f3a56dc3f6041e49ebd5692f8bb2b821"
server_key_iv: "0beb6fd36936db98787b0cc67e4c73e7"
server_key_version: "encryptionkey@0"
I20260619 04:50:10.477384 21727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:10.491006 21727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:10.491712 21727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:10.491999 21727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:10.510092 21727 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:33147
I20260619 04:50:10.510099 21792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:33147 every 8 connection(s)
I20260619 04:50:10.511402 21727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:10.513094 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 21727
I20260619 04:50:10.513281 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:10.513747 4046 external_mini_cluster.cc:1428] Setting key d98f47e9dc2e3463c1ff4305a198920b
I20260619 04:50:10.515550 21793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:10.527079 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:10.515559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.3.243.190:33147)
0619 04:50:10.516609 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:10.516619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:10.516659 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:50:10.518181 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:10.518201 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:10.518654 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:10.518970 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:10.519588 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:10.520968 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:10.522822 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:10.523441 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:10.523556 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:10.524873 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:10.524880 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:10.524896 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:10.525444 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:10.525521 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:10.525541 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:10.525636 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:10.525862 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 04:50:10.526203 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":383,"threads_started":1}
I20260619 04:50:10.528458 21793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585: Bootstrap starting.
I20260619 04:50:10.531611 21793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:10.532621 21793 log.cc:824] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:10.535400 21793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585: No bootstrap required, opened a new log
I20260619 04:50:10.542573 21793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5425920e5b45398cdc77ef6f3ef585" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33147 } }
I20260619 04:50:10.542960 21793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:10.543072 21793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5425920e5b45398cdc77ef6f3ef585, State: Initialized, Role: FOLLOWER
I20260619 04:50:10.543704 21793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5425920e5b45398cdc77ef6f3ef585" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33147 } }
I20260619 04:50:10.543978 21793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:10.544173 21793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:10.544370 21793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:10.546723 21793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5425920e5b45398cdc77ef6f3ef585" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33147 } }
I20260619 04:50:10.547227 21793 leader_election.cc:302] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c5425920e5b45398cdc77ef6f3ef585; no voters:
I20260619 04:50:10.547791 21793 leader_election.cc:288] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:10.547935 21798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:10.548732 21798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5425920e5b45398cdc77ef6f3ef585, State: Running, Role: LEADER
I20260619 04:50:10.549256 21798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5425920e5b45398cdc77ef6f3ef585" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33147 } }
I20260619 04:50:10.549878 21793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:10.552062 21799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c5425920e5b45398cdc77ef6f3ef585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5425920e5b45398cdc77ef6f3ef585" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33147 } } }
I20260619 04:50:10.552350 21800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c5425920e5b45398cdc77ef6f3ef585. Latest consensus state: current_term: 1 leader_uuid: "3c5425920e5b45398cdc77ef6f3ef585" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c5425920e5b45398cdc77ef6f3ef585" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33147 } } }
I20260619 04:50:10.552567 21799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:10.552606 21800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:10.555197 21806 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:10.560360 21806 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:10.569252 21806 catalog_manager.cc:1347] Generated new cluster ID: 8aa3aee12088493bb5e4b12d041ceb23
I20260619 04:50:10.569387 21806 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:10.577656 21806 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:10.579396 21806 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:10.591346 21806 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585: Generated new TSK 0
I20260619 04:50:10.592422 21806 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:10.598196 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:33147
--builtin_ntp_servers=127.3.243.148:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:50:10.926245 21817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:10.926563 21817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:10.926721 21817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:10.933810 21817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:10.933951 21817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:10.934162 21817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:50:10.943256 21817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:33147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:10.944674 21817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:10.946470 21817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:10.960626 21822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:10.960697 21823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:10.960876 21825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:10.961982 21817 server_base.cc:1034] running on GCE node
I20260619 04:50:10.962606 21817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:10.963578 21817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:10.964826 21817 hybrid_clock.cc:648] HybridClock initialized: now 1781844610964775 us; error 67 us; skew 500 ppm
I20260619 04:50:10.965160 21817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:10.967463 21817 webserver.cc:460] Webserver started at http://127.3.243.129:40227/ using document root <none> and password file <none>
I20260619 04:50:10.968267 21817 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:10.968402 21817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:10.968715 21817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:10.971524 21817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "555d336118cc4441882710238a632959"
format_stamp: "Formatted at 2026-06-19 04:50:10 on dist-test-slave-2qp8"
server_key: "1c35f2f890f36e0921e40b564bdbeabc"
server_key_iv: "90599f578ee38e87da6b8fc57efc8650"
server_key_version: "encryptionkey@0"
I20260619 04:50:10.972416 21817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "555d336118cc4441882710238a632959"
format_stamp: "Formatted at 2026-06-19 04:50:10 on dist-test-slave-2qp8"
server_key: "1c35f2f890f36e0921e40b564bdbeabc"
server_key_iv: "90599f578ee38e87da6b8fc57efc8650"
server_key_version: "encryptionkey@0"
I20260619 04:50:10.977607 21817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:50:10.981093 21831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:10.982599 21817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:10.982781 21817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "555d336118cc4441882710238a632959"
format_stamp: "Formatted at 2026-06-19 04:50:10 on dist-test-slave-2qp8"
server_key: "1c35f2f890f36e0921e40b564bdbeabc"
server_key_iv: "90599f578ee38e87da6b8fc57efc8650"
server_key_version: "encryptionkey@0"
I20260619 04:50:10.983000 21817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:10.996757 21817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:10.997547 21817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:10.997862 21817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:10.998795 21817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:11.000281 21817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:11.000373 21817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:11.000521 21817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:11.000625 21817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:11.032523 21817 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:34959
I20260619 04:50:11.032578 21943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:34959 every 8 connection(s)
I20260619 04:50:11.033900 21817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:50:11.035480 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 21817
I20260619 04:50:11.035650 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:50:11.036100 4046 external_mini_cluster.cc:1428] Setting key 361fd8d2bad944230bce217c61f1c096
I20260619 04:50:11.039481 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:33147
--builtin_ntp_servers=127.3.243.148:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:50:11.049075 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:11.036632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:36045 (local address 127.3.243.190:33147)
0619 04:50:11.037141 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:11.037147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:11.038544 (+ 1397us) server_negotiation.cc:407] Connection header received
0619 04:50:11.041377 (+ 2833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:11.041381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:11.041442 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:11.041524 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:11.042854 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.043566 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.046128 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.046543 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.046634 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:11.048233 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:11.048238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:11.048241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:11.048260 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:11.048305 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:11.048317 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:11.048417 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:11.048642 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 04:50:11.048733 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":259,"threads_started":1}
I20260619 04:50:11.050107 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:11.037280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:33147 (local address 127.3.243.129:36045)
0619 04:50:11.038369 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:11.038425 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:11.041075 (+ 2650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:11.041783 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:11.041797 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:11.041928 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:11.042696 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:11.042720 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.043731 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.043739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:11.045979 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:11.045997 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.046762 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.046770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:11.047101 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:11.047847 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:50:11.047853 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:11.047918 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:50:11.048019 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:11.048419 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:11.048433 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:11.048442 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 04:50:11.048615 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 04:50:11.048981 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":294,"threads_started":1}
I20260619 04:50:11.051898 21944 heartbeater.cc:347] Connected to a master server at 127.3.243.190:33147
I20260619 04:50:11.052245 21944 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:11.052973 21944 heartbeater.cc:511] Master 127.3.243.190:33147 requested a full tablet report, sending...
I20260619 04:50:11.055183 21758 ts_manager.cc:194] Registered new tserver with Master: 555d336118cc4441882710238a632959 (127.3.243.129:34959)
I20260619 04:50:11.057175 21758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.129:36045
W20260619 04:50:11.364884 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:11.365144 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:11.365229 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:11.372351 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:11.372444 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:11.372560 21948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:50:11.381481 21948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:33147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:11.382786 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:11.384642 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:11.398273 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:11.398360 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:11.398325 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:11.401710 21948 server_base.cc:1034] running on GCE node
I20260619 04:50:11.402359 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:11.403332 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:11.404563 21948 hybrid_clock.cc:648] HybridClock initialized: now 1781844611404529 us; error 56 us; skew 500 ppm
I20260619 04:50:11.404901 21948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:11.407253 21948 webserver.cc:460] Webserver started at http://127.3.243.130:38585/ using document root <none> and password file <none>
I20260619 04:50:11.407963 21948 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:11.408097 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:11.408464 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:11.411347 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "24b2a944d548439eafc888174f490304"
format_stamp: "Formatted at 2026-06-19 04:50:11 on dist-test-slave-2qp8"
server_key: "7aa384f081f39687405a571c4c119054"
server_key_iv: "1f1a60b2a3dad90a19e7bd25a1d16693"
server_key_version: "encryptionkey@0"
I20260619 04:50:11.412209 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "24b2a944d548439eafc888174f490304"
format_stamp: "Formatted at 2026-06-19 04:50:11 on dist-test-slave-2qp8"
server_key: "7aa384f081f39687405a571c4c119054"
server_key_iv: "1f1a60b2a3dad90a19e7bd25a1d16693"
server_key_version: "encryptionkey@0"
I20260619 04:50:11.419592 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260619 04:50:11.423790 21962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:11.425262 21948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 04:50:11.425467 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "24b2a944d548439eafc888174f490304"
format_stamp: "Formatted at 2026-06-19 04:50:11 on dist-test-slave-2qp8"
server_key: "7aa384f081f39687405a571c4c119054"
server_key_iv: "1f1a60b2a3dad90a19e7bd25a1d16693"
server_key_version: "encryptionkey@0"
I20260619 04:50:11.425719 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:11.454962 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:11.456044 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:11.456409 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:11.457505 21948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:11.459048 21948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:11.459184 21948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:11.459354 21948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:11.459456 21948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:11.493269 21948 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:33581
I20260619 04:50:11.493340 22074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:33581 every 8 connection(s)
I20260619 04:50:11.494699 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:50:11.498291 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 21948
I20260619 04:50:11.498500 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:50:11.499140 4046 external_mini_cluster.cc:1428] Setting key 5089aedaabd9bcad6a707d36663bba7e
I20260619 04:50:11.502427 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:33147
--builtin_ntp_servers=127.3.243.148:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:50:11.508972 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:11.497396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:34555 (local address 127.3.243.190:33147)
0619 04:50:11.497631 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:11.497638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:11.499303 (+ 1665us) server_negotiation.cc:407] Connection header received
0619 04:50:11.501439 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:11.501445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:11.501532 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:11.501662 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:11.502774 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.504102 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.505694 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.506498 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.506653 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:11.507949 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:11.507958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:11.507966 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:11.507994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:11.508039 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:11.508059 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:11.508206 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:11.508422 (+ 216us) server_negotiation.cc:299] Negotiation successful
0619 04:50:11.508540 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 04:50:11.509804 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:11.498106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:33147 (local address 127.3.243.130:34555)
0619 04:50:11.499120 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:11.499162 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:11.501121 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:11.501845 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:11.501853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:11.501941 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:11.502550 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:11.502586 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.504281 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.504285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:11.505530 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:11.505542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.506699 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.506704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:11.506900 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:11.507615 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:50:11.507617 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:11.507660 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:50:11.507736 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:11.508199 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:11.508207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:11.508212 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:50:11.508369 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 04:50:11.508649 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":363,"threads_started":1}
I20260619 04:50:11.511279 22075 heartbeater.cc:347] Connected to a master server at 127.3.243.190:33147
I20260619 04:50:11.511623 22075 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:11.512593 22075 heartbeater.cc:511] Master 127.3.243.190:33147 requested a full tablet report, sending...
I20260619 04:50:11.514182 21758 ts_manager.cc:194] Registered new tserver with Master: 24b2a944d548439eafc888174f490304 (127.3.243.130:33581)
I20260619 04:50:11.515482 21758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.130:34555
W20260619 04:50:11.827109 22078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:11.827390 22078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:11.827476 22078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:11.834789 22078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:11.834892 22078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:11.835000 22078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:50:11.843781 22078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:33147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:11.845072 22078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:11.846760 22078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:11.859854 22083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:11.859907 22084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:11.859944 22086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:11.860965 22078 server_base.cc:1034] running on GCE node
I20260619 04:50:11.861539 22078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:11.862462 22078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:11.863674 22078 hybrid_clock.cc:648] HybridClock initialized: now 1781844611863627 us; error 42 us; skew 500 ppm
I20260619 04:50:11.864068 22078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:11.866297 22078 webserver.cc:460] Webserver started at http://127.3.243.131:33373/ using document root <none> and password file <none>
I20260619 04:50:11.866973 22078 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:11.867090 22078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:11.867411 22078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:11.870288 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "5846b08e0d39430f93d7652113526af2"
format_stamp: "Formatted at 2026-06-19 04:50:11 on dist-test-slave-2qp8"
server_key: "1240c99a1de528dfe4405f8e9122f920"
server_key_iv: "012aed8f5b02914e438c66d8ae2301f5"
server_key_version: "encryptionkey@0"
I20260619 04:50:11.871063 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "5846b08e0d39430f93d7652113526af2"
format_stamp: "Formatted at 2026-06-19 04:50:11 on dist-test-slave-2qp8"
server_key: "1240c99a1de528dfe4405f8e9122f920"
server_key_iv: "012aed8f5b02914e438c66d8ae2301f5"
server_key_version: "encryptionkey@0"
I20260619 04:50:11.876274 22078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 04:50:11.879738 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:11.881317 22078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 04:50:11.881489 22078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "5846b08e0d39430f93d7652113526af2"
format_stamp: "Formatted at 2026-06-19 04:50:11 on dist-test-slave-2qp8"
server_key: "1240c99a1de528dfe4405f8e9122f920"
server_key_iv: "012aed8f5b02914e438c66d8ae2301f5"
server_key_version: "encryptionkey@0"
I20260619 04:50:11.881704 22078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:11.906594 22078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:11.907459 22078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:11.907778 22078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:11.908812 22078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:11.910434 22078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:11.910548 22078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:11.910663 22078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:11.910744 22078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:11.944620 22078 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:39583
I20260619 04:50:11.944711 22204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:39583 every 8 connection(s)
I20260619 04:50:11.945940 22078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:50:11.950832 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 22078
I20260619 04:50:11.951014 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:50:11.951628 4046 external_mini_cluster.cc:1428] Setting key 386ae3b037cf02f5ce6a75a4bb08d30a
I20260619 04:50:11.959033 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:11.948538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:60129 (local address 127.3.243.190:33147)
0619 04:50:11.948726 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:11.948730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:11.950122 (+ 1392us) server_negotiation.cc:407] Connection header received
0619 04:50:11.952608 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:11.952613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:11.952710 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:11.952826 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:11.953874 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.954672 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.956475 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.956872 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.957004 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:11.958217 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:11.958221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:11.958224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:11.958242 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:11.958288 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:11.958299 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:11.958403 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:11.958608 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 04:50:11.958698 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:50:11.959959 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:11.949057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:33147 (local address 127.3.243.131:60129)
0619 04:50:11.949954 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:11.949993 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:11.952344 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:11.953047 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:11.953054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:11.953152 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:11.953690 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:11.953706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.954841 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.954845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:11.956341 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:11.956353 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.957082 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.957089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:11.957302 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:11.957897 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:50:11.957900 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:11.957934 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:50:11.958006 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:11.958403 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:11.958415 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:11.958425 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 04:50:11.958599 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 04:50:11.958988 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":265,"threads_started":1}
I20260619 04:50:11.961251 22205 heartbeater.cc:347] Connected to a master server at 127.3.243.190:33147
I20260619 04:50:11.961541 22205 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:11.962204 22205 heartbeater.cc:511] Master 127.3.243.190:33147 requested a full tablet report, sending...
I20260619 04:50:11.963492 21758 ts_manager.cc:194] Registered new tserver with Master: 5846b08e0d39430f93d7652113526af2 (127.3.243.131:39583)
I20260619 04:50:11.964340 21758 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.131:60129
I20260619 04:50:11.966611 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:50:11.976537 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:11.970672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.3.243.190:33147)
0619 04:50:11.970868 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:11.970872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:11.971388 (+ 516us) server_negotiation.cc:407] Connection header received
0619 04:50:11.971521 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:11.971524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:11.971589 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:11.971678 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:11.972298 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.972950 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.974330 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:11.974680 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:11.974762 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:11.975811 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:11.975816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:11.975828 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:11.975844 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:11.975873 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:11.975884 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:11.975949 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:11.976089 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 04:50:11.976286 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:50:11.980300 21758 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:50:11.987596 21758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:50:12.016978 22217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:11.999725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:39583 (local address 127.0.0.1:56018)
0619 04:50:12.007455 (+ 7730us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:12.007505 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:12.007685 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:12.008945 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:12.008956 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:12.008999 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:12.009437 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.009453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.011604 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.011609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.014191 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.014205 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.015707 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.015713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.016200 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.016280 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 04:50:12.016406 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 04:50:12.016594 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7454,"mutex_wait_us":1,"thread_start_us":7324,"threads_started":1}
I20260619 04:50:12.017100 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:11.999887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.3.243.131:39583)
0619 04:50:12.007445 (+ 7558us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:12.007454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:12.007920 (+ 466us) server_negotiation.cc:407] Connection header received
0619 04:50:12.008052 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:12.008061 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:12.008364 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:12.008551 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:12.009719 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.011448 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.014391 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.015489 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.015634 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.015942 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:12.016536 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 04:50:12.016711 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7377,"thread_start_us":6758,"threads_started":1}
I20260619 04:50:12.020259 22221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.000537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34959 (local address 127.0.0.1:33772)
0619 04:50:12.010256 (+ 9719us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:12.010281 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:12.010416 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:12.012563 (+ 2147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:12.012568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:12.012593 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:12.012982 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.012994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.014731 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.014738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.017614 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.017629 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.019334 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.019341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.019506 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.019556 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 04:50:12.019652 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 04:50:12.019780 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9535,"mutex_wait_us":124,"thread_start_us":10262,"threads_started":1}
I20260619 04:50:12.020648 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.008501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.3.243.130:33581)
0619 04:50:12.012541 (+ 4040us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:12.012549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:12.012573 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:50:12.012660 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:12.012665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:12.012811 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:12.012998 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:12.013560 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.014753 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.018473 (+ 3720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.019518 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.019677 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.019912 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:12.020147 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 04:50:12.020339 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3892,"thread_start_us":3793,"threads_started":1}
I20260619 04:50:12.020807 22219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.008165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:33581 (local address 127.0.0.1:55544)
0619 04:50:12.011079 (+ 2914us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:12.011101 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:12.011219 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:12.012987 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:12.012992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:12.013016 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:12.013391 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.013399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.014908 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.014913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.018294 (+ 3381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.018305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.019740 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.019746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.019918 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.019947 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 04:50:12.020058 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 04:50:12.020221 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2778,"mutex_wait_us":27,"thread_start_us":2333,"threads_started":1}
I20260619 04:50:12.021386 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.000667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.3.243.129:34959)
0619 04:50:12.011699 (+ 11032us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:12.011708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:12.011736 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:50:12.011846 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:12.011854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:12.012095 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:12.012349 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:12.013188 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.014603 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.017820 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.019124 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.019281 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.019522 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:12.020849 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0619 04:50:12.021133 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10883,"thread_start_us":10791,"threads_started":1}
I20260619 04:50:12.021494 22140 tablet_service.cc:1467] Processing CreateTablet for tablet f0b03205846147e78475a6b37fbb76b9 (DEFAULT_TABLE table=test-table [id=c38c3d0d59d64fb8b9cabf4ac98c375e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:12.024852 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b03205846147e78475a6b37fbb76b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:12.024782 22010 tablet_service.cc:1467] Processing CreateTablet for tablet f0b03205846147e78475a6b37fbb76b9 (DEFAULT_TABLE table=test-table [id=c38c3d0d59d64fb8b9cabf4ac98c375e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:12.024791 21879 tablet_service.cc:1467] Processing CreateTablet for tablet f0b03205846147e78475a6b37fbb76b9 (DEFAULT_TABLE table=test-table [id=c38c3d0d59d64fb8b9cabf4ac98c375e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:12.026161 22010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b03205846147e78475a6b37fbb76b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:12.026160 21879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b03205846147e78475a6b37fbb76b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:12.042606 22225 tablet_bootstrap.cc:492] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: Bootstrap starting.
I20260619 04:50:12.047199 22223 tablet_bootstrap.cc:492] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: Bootstrap starting.
I20260619 04:50:12.049714 22224 tablet_bootstrap.cc:492] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: Bootstrap starting.
I20260619 04:50:12.050541 22225 tablet_bootstrap.cc:654] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:12.050709 22223 tablet_bootstrap.cc:654] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:12.051668 22223 log.cc:824] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:12.051836 22225 log.cc:824] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:12.053593 22224 tablet_bootstrap.cc:654] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:12.054641 22223 tablet_bootstrap.cc:492] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: No bootstrap required, opened a new log
I20260619 04:50:12.054965 22223 ts_tablet_manager.cc:1397] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 04:50:12.055019 22224 log.cc:824] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:12.059999 21944 heartbeater.cc:503] Master 127.3.243.190:33147 was elected leader, sending a full tablet report...
I20260619 04:50:12.061139 22225 tablet_bootstrap.cc:492] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: No bootstrap required, opened a new log
I20260619 04:50:12.061393 22224 tablet_bootstrap.cc:492] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: No bootstrap required, opened a new log
I20260619 04:50:12.061431 22225 ts_tablet_manager.cc:1397] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.003s
I20260619 04:50:12.062108 22224 ts_tablet_manager.cc:1397] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260619 04:50:12.063721 22223 raft_consensus.cc:348] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } }
I20260619 04:50:12.064255 22223 raft_consensus.cc:374] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:12.064357 22223 raft_consensus.cc:729] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5846b08e0d39430f93d7652113526af2, State: Initialized, Role: FOLLOWER
I20260619 04:50:12.065071 22223 consensus_queue.cc:260] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } }
I20260619 04:50:12.066126 22205 heartbeater.cc:503] Master 127.3.243.190:33147 was elected leader, sending a full tablet report...
I20260619 04:50:12.066792 22223 ts_tablet_manager.cc:1428] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260619 04:50:12.071308 22225 raft_consensus.cc:348] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } }
I20260619 04:50:12.071642 22225 raft_consensus.cc:374] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:12.071735 22225 raft_consensus.cc:729] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555d336118cc4441882710238a632959, State: Initialized, Role: FOLLOWER
I20260619 04:50:12.071928 22224 raft_consensus.cc:348] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } }
I20260619 04:50:12.072376 22224 raft_consensus.cc:374] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:12.072484 22224 raft_consensus.cc:729] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b2a944d548439eafc888174f490304, State: Initialized, Role: FOLLOWER
I20260619 04:50:12.072492 22225 consensus_queue.cc:260] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } }
I20260619 04:50:12.073074 22224 consensus_queue.cc:260] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } }
I20260619 04:50:12.074165 22225 ts_tablet_manager.cc:1428] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260619 04:50:12.074194 22075 heartbeater.cc:503] Master 127.3.243.190:33147 was elected leader, sending a full tablet report...
I20260619 04:50:12.074826 22224 ts_tablet_manager.cc:1428] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260619 04:50:12.197661 22206 tablet.cc:2369] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:50:12.246907 22076 tablet.cc:2369] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:50:12.286309 21945 tablet.cc:2369] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:12.346078 22229 raft_consensus.cc:482] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:50:12.346414 22229 raft_consensus.cc:504] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } }
I20260619 04:50:12.348162 22229 leader_election.cc:288] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24b2a944d548439eafc888174f490304 (127.3.243.130:33581), 555d336118cc4441882710238a632959 (127.3.243.129:34959)
I20260619 04:50:12.354679 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.348616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:47451 (local address 127.3.243.130:33581)
0619 04:50:12.348836 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:12.348843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:12.348862 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:50:12.348937 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:12.348941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:12.349062 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:12.349160 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:12.349896 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.350933 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.352904 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.353835 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.353966 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.354015 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:12.354325 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 04:50:12.354415 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:50:12.354887 22207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.348370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:33581 (local address 127.3.243.131:47451)
0619 04:50:12.348563 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:12.348590 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:12.348728 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:12.349373 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:12.349376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:12.349450 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:12.349731 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.349743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.351055 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.351061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.352752 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.352763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.353975 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.353981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.354159 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.354194 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 04:50:12.354291 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 04:50:12.354430 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":42}
I20260619 04:50:12.355386 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.349037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:49749 (local address 127.3.243.129:34959)
0619 04:50:12.349246 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:12.349252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:12.349564 (+ 312us) server_negotiation.cc:407] Connection header received
0619 04:50:12.349769 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:12.349775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:12.349857 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:12.349945 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:12.350649 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.351558 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.353421 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.354432 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.354580 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.354644 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:12.354991 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 04:50:12.355113 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260619 04:50:12.355422 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.348691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:34959 (local address 127.3.243.131:49749)
0619 04:50:12.349441 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:12.349454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:12.349621 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:12.349987 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:12.349993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:12.350018 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:12.350388 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.350400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.351701 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.351705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.353249 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:12.353262 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.354610 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.354616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:12.354802 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.354849 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 04:50:12.354965 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 04:50:12.355086 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":1,"thread_start_us":417,"threads_started":1}
I20260619 04:50:12.355957 22030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b03205846147e78475a6b37fbb76b9" candidate_uuid: "5846b08e0d39430f93d7652113526af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b2a944d548439eafc888174f490304" is_pre_election: true
I20260619 04:50:12.356550 22030 raft_consensus.cc:2454] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5846b08e0d39430f93d7652113526af2 in term 0.
I20260619 04:50:12.356530 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b03205846147e78475a6b37fbb76b9" candidate_uuid: "5846b08e0d39430f93d7652113526af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555d336118cc4441882710238a632959" is_pre_election: true
I20260619 04:50:12.357141 21899 raft_consensus.cc:2454] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5846b08e0d39430f93d7652113526af2 in term 0.
I20260619 04:50:12.357287 22094 leader_election.cc:302] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2a944d548439eafc888174f490304, 5846b08e0d39430f93d7652113526af2; no voters:
I20260619 04:50:12.357725 22229 raft_consensus.cc:2781] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:50:12.357872 22229 raft_consensus.cc:482] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:50:12.358000 22229 raft_consensus.cc:3037] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:12.360176 22229 raft_consensus.cc:504] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } }
I20260619 04:50:12.360821 22229 leader_election.cc:288] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [CANDIDATE]: Term 1 election: Requested vote from peers 24b2a944d548439eafc888174f490304 (127.3.243.130:33581), 555d336118cc4441882710238a632959 (127.3.243.129:34959)
I20260619 04:50:12.361186 22030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b03205846147e78475a6b37fbb76b9" candidate_uuid: "5846b08e0d39430f93d7652113526af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b2a944d548439eafc888174f490304"
I20260619 04:50:12.361346 21899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b03205846147e78475a6b37fbb76b9" candidate_uuid: "5846b08e0d39430f93d7652113526af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555d336118cc4441882710238a632959"
I20260619 04:50:12.361397 22030 raft_consensus.cc:3037] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:12.361549 21899 raft_consensus.cc:3037] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:12.364681 22030 raft_consensus.cc:2454] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5846b08e0d39430f93d7652113526af2 in term 1.
I20260619 04:50:12.364670 21899 raft_consensus.cc:2454] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5846b08e0d39430f93d7652113526af2 in term 1.
I20260619 04:50:12.365091 22093 leader_election.cc:302] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555d336118cc4441882710238a632959, 5846b08e0d39430f93d7652113526af2; no voters:
I20260619 04:50:12.365412 22229 raft_consensus.cc:2781] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:12.365948 22229 raft_consensus.cc:686] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 1 LEADER]: Becoming Leader. State: Replica: 5846b08e0d39430f93d7652113526af2, State: Running, Role: LEADER
I20260619 04:50:12.366412 22229 consensus_queue.cc:237] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } }
I20260619 04:50:12.371232 21757 catalog_manager.cc:5495] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5846b08e0d39430f93d7652113526af2 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "5846b08e0d39430f93d7652113526af2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5846b08e0d39430f93d7652113526af2" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 } health_report { overall_health: UNKNOWN } } }
I20260619 04:50:12.466535 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.461370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.3.243.131:39583)
0619 04:50:12.461538 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:12.461543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:12.461560 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:50:12.461712 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:12.461716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:12.461791 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:12.461915 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:50:12.462274 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.463006 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.464553 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.464939 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.465025 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.465491 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:50:12.465970 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:50:12.466079 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:12.466194 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 04:50:12.466251 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:50:12.474346 22030 raft_consensus.cc:1261] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [term 1 FOLLOWER]: Refusing update from remote peer 5846b08e0d39430f93d7652113526af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:12.474378 21899 raft_consensus.cc:1261] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [term 1 FOLLOWER]: Refusing update from remote peer 5846b08e0d39430f93d7652113526af2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:12.475308 22233 consensus_queue.cc:1035] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b2a944d548439eafc888174f490304" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:12.475638 22229 consensus_queue.cc:1035] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555d336118cc4441882710238a632959" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 34959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:12.487057 22236 mvcc.cc:205] Tried to move back new op lower bound from 7298435532685266944 to 7298435532256829440. Current Snapshot: MvccSnapshot[applied={T|T < 7298435532685266944}]
I20260619 04:50:12.502049 21757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:12.502378 21757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:12.506151 21757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c5425920e5b45398cdc77ef6f3ef585: Sending DeleteTablet for 3 replicas of tablet f0b03205846147e78475a6b37fbb76b9
I20260619 04:50:12.507349 22140 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b03205846147e78475a6b37fbb76b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:12 UTC) from {username='slave'} at 127.0.0.1:56018
I20260619 04:50:12.507418 21879 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b03205846147e78475a6b37fbb76b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:12 UTC) from {username='slave'} at 127.0.0.1:33772
I20260619 04:50:12.507608 22010 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b03205846147e78475a6b37fbb76b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:12 UTC) from {username='slave'} at 127.0.0.1:55544
I20260619 04:50:12.509625 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 21817
I20260619 04:50:12.518507 22248 tablet_replica.cc:330] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: stopping tablet replica
I20260619 04:50:12.518919 22248 raft_consensus.cc:2229] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:12.519423 22248 raft_consensus.cc:2258] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:12.523540 22248 ts_tablet_manager.cc:1905] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:12.533516 22247 tablet_replica.cc:330] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: stopping tablet replica
I20260619 04:50:12.534118 22247 raft_consensus.cc:2229] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:12.534711 22247 raft_consensus.cc:2258] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:12.536875 22248 ts_tablet_manager.cc:1918] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:12.537024 22248 log.cc:1196] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/f0b03205846147e78475a6b37fbb76b9
I20260619 04:50:12.537014 22247 ts_tablet_manager.cc:1905] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:12.537520 22248 ts_tablet_manager.cc:1939] T f0b03205846147e78475a6b37fbb76b9 P 24b2a944d548439eafc888174f490304: Deleting consensus metadata
I20260619 04:50:12.539227 21743 catalog_manager.cc:4841] TS 24b2a944d548439eafc888174f490304 (127.3.243.130:33581): tablet f0b03205846147e78475a6b37fbb76b9 (table test-table [id=c38c3d0d59d64fb8b9cabf4ac98c375e]) successfully deleted
I20260619 04:50:12.544013 22247 ts_tablet_manager.cc:1918] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:12.544206 22247 log.cc:1196] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/f0b03205846147e78475a6b37fbb76b9
I20260619 04:50:12.544739 22247 ts_tablet_manager.cc:1939] T f0b03205846147e78475a6b37fbb76b9 P 555d336118cc4441882710238a632959: Deleting consensus metadata
I20260619 04:50:12.546541 21742 catalog_manager.cc:4841] TS 555d336118cc4441882710238a632959 (127.3.243.129:34959): tablet f0b03205846147e78475a6b37fbb76b9 (table test-table [id=c38c3d0d59d64fb8b9cabf4ac98c375e]) successfully deleted
I20260619 04:50:12.549952 22246 tablet_replica.cc:330] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: stopping tablet replica
I20260619 04:50:12.550400 22246 raft_consensus.cc:2229] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:50:12.550983 22246 raft_consensus.cc:2258] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:12.555260 22246 ts_tablet_manager.cc:1905] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:12.556634 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.525553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.3.243.129:34959)
0619 04:50:12.525795 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:12.525803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:12.546224 (+ 20421us) server_negotiation.cc:407] Connection header received
0619 04:50:12.546424 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:12.546429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:12.546540 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:12.546648 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:12.547196 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.548281 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.551562 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.552178 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.552264 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.554980 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:12.554990 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:12.555005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:12.555750 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:12.555803 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:12.555825 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:12.555901 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:12.556195 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 04:50:12.556316 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:50:12.557936 21939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:12.563005 22246 ts_tablet_manager.cc:1918] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:12.563221 22246 log.cc:1196] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/f0b03205846147e78475a6b37fbb76b9
I20260619 04:50:12.563838 22246 ts_tablet_manager.cc:1939] T f0b03205846147e78475a6b37fbb76b9 P 5846b08e0d39430f93d7652113526af2: Deleting consensus metadata
I20260619 04:50:12.565730 21743 catalog_manager.cc:4841] TS 5846b08e0d39430f93d7652113526af2 (127.3.243.131:39583): tablet f0b03205846147e78475a6b37fbb76b9 (table test-table [id=c38c3d0d59d64fb8b9cabf4ac98c375e]) successfully deleted
I20260619 04:50:12.649641 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 21817
I20260619 04:50:12.667801 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 21948
I20260619 04:50:12.674762 22222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.668616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55546 (local address 127.3.243.130:33581)
0619 04:50:12.668790 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:12.668798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:12.668819 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:50:12.669046 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:12.669052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:12.669158 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:12.669293 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:12.669665 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.670363 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.671591 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.672009 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.672191 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.673453 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:12.673470 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:12.673485 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:12.674054 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:12.674103 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:12.674121 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:12.674261 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:12.674346 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 04:50:12.674444 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:50:12.675391 22070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:12.809338 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 21948
I20260619 04:50:12.826258 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 22078
I20260619 04:50:12.833573 22218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:12.827079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.3.243.131:39583)
0619 04:50:12.827279 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:12.827290 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:12.827312 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:50:12.827527 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:12.827538 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:12.827629 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:12.827755 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:12.828190 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.828969 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.830334 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:12.830744 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:12.830904 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:12.832042 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:12.832051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:12.832067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:12.832779 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:12.832826 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:12.832845 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:12.832982 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:12.833118 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 04:50:12.833222 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:50:12.834270 22200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:12.960516 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 22078
I20260619 04:50:12.976181 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 21727
I20260619 04:50:12.977286 21788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:13.086282 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 21727
2026-06-19T04:50:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3070 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:13 dist-test-slave-2qp8 krb5kdc[22256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:13 dist-test-slave-2qp8 krb5kdc[22256](info): set up 2 sockets
Jun 19 04:50:13 dist-test-slave-2qp8 krb5kdc[22256](info): commencing operation
krb5kdc: starting...
W20260619 04:50:14.204399 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:14 dist-test-slave-2qp8 krb5kdc[22256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:50:14.455875 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:40829
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:40829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:14.777828 22269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:14.778095 22269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:14.778180 22269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:14.785089 22269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:14.785195 22269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:14.785240 22269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:14.785296 22269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:14.785339 22269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:14.794646 22269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:40829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:40829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:14.795832 22269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:14.797180 22269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:14.809415 22275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:14.809496 22277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:14.809564 22274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:14.810199 22269 server_base.cc:1034] running on GCE node
I20260619 04:50:14.810942 22269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:14.812361 22269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:14.813587 22269 hybrid_clock.cc:648] HybridClock initialized: now 1781844614813543 us; error 59 us; skew 500 ppm
I20260619 04:50:14.813930 22269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:14 dist-test-slave-2qp8 krb5kdc[22256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:14.817049 22269 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:50:14.818436 22269 webserver.cc:460] Webserver started at http://127.3.243.190:39489/ using document root <none> and password file <none>
I20260619 04:50:14.819089 22269 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:14.819231 22269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:14.819533 22269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:14.822455 22269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "d1cf32a430f24a779e6239b42f72bdb3"
format_stamp: "Formatted at 2026-06-19 04:50:14 on dist-test-slave-2qp8"
server_key: "3b266f91ec53b9f9dc2616829ab71c08"
server_key_iv: "333be6dd627635df94174cf03894e97b"
server_key_version: "encryptionkey@0"
I20260619 04:50:14.823197 22269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "d1cf32a430f24a779e6239b42f72bdb3"
format_stamp: "Formatted at 2026-06-19 04:50:14 on dist-test-slave-2qp8"
server_key: "3b266f91ec53b9f9dc2616829ab71c08"
server_key_iv: "333be6dd627635df94174cf03894e97b"
server_key_version: "encryptionkey@0"
I20260619 04:50:14.828434 22269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 04:50:14.831614 22284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:14.832825 22269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:50:14.832999 22269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "d1cf32a430f24a779e6239b42f72bdb3"
format_stamp: "Formatted at 2026-06-19 04:50:14 on dist-test-slave-2qp8"
server_key: "3b266f91ec53b9f9dc2616829ab71c08"
server_key_iv: "333be6dd627635df94174cf03894e97b"
server_key_version: "encryptionkey@0"
I20260619 04:50:14.833189 22269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:14.887219 22269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:14.892020 22269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:14.892345 22269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:14.911540 22269 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:40829
I20260619 04:50:14.911558 22335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:40829 every 8 connection(s)
I20260619 04:50:14.913057 22269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:14.916819 22336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:14.920146 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 22269
I20260619 04:50:14.920331 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:14.920900 4046 external_mini_cluster.cc:1428] Setting key 110c45bbc67993d3f60c3ca8b09d3622
I20260619 04:50:14.929178 22336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3: Bootstrap starting.
I20260619 04:50:14.932089 22336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:14.933100 22336 log.cc:824] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3: Log is configured to *not* fsync() on all Append() calls
Jun 19 04:50:14 dist-test-slave-2qp8 krb5kdc[22256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:14.936422 22336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3: No bootstrap required, opened a new log
I20260619 04:50:14.943842 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:14.922969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.3.243.190:40829)
0619 04:50:14.924003 (+ 1034us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:14.924018 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:14.924070 (+ 52us) server_negotiation.cc:407] Connection header received
0619 04:50:14.925782 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:14.925802 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:14.926192 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:14.926587 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:14.927968 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:14.929332 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:14.930735 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:14.931406 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:14.931590 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:14.936394 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:14.936412 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:14.936426 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:14.936481 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:14.939692 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:14.940572 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:14.940578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:14.940583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:14.940751 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:14.941166 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:14.941169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:14.941171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:14.941637 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:14.941879 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:14.942390 (+ 511us) server_negotiation.cc:299] Negotiation successful
0619 04:50:14.942834 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":442,"threads_started":1}
I20260619 04:50:14.945427 22336 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cf32a430f24a779e6239b42f72bdb3" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40829 } }
I20260619 04:50:14.945874 22336 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:14.945981 22336 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1cf32a430f24a779e6239b42f72bdb3, State: Initialized, Role: FOLLOWER
I20260619 04:50:14.946596 22336 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cf32a430f24a779e6239b42f72bdb3" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40829 } }
I20260619 04:50:14.946945 22336 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:14.947080 22336 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:14.947197 22336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:14.949442 22336 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cf32a430f24a779e6239b42f72bdb3" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40829 } }
I20260619 04:50:14.950062 22336 leader_election.cc:302] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1cf32a430f24a779e6239b42f72bdb3; no voters:
I20260619 04:50:14.950668 22336 leader_election.cc:288] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:14.950795 22341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:14.951582 22341 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [term 1 LEADER]: Becoming Leader. State: Replica: d1cf32a430f24a779e6239b42f72bdb3, State: Running, Role: LEADER
I20260619 04:50:14.952155 22341 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cf32a430f24a779e6239b42f72bdb3" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40829 } }
I20260619 04:50:14.952893 22336 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:14.955490 22343 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1cf32a430f24a779e6239b42f72bdb3. Latest consensus state: current_term: 1 leader_uuid: "d1cf32a430f24a779e6239b42f72bdb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cf32a430f24a779e6239b42f72bdb3" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40829 } } }
I20260619 04:50:14.957139 22343 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:14.956161 22342 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1cf32a430f24a779e6239b42f72bdb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1cf32a430f24a779e6239b42f72bdb3" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40829 } } }
I20260619 04:50:14.957728 22342 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:14.965252 22350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:14.970908 22350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:14.980142 22350 catalog_manager.cc:1347] Generated new cluster ID: a3b15e11a0d0450d90e6225365652064
I20260619 04:50:14.980309 22350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:14.998917 22350 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:15.000339 22350 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:15.021502 22350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3: Generated new TSK 0
I20260619 04:50:15.022403 22350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:50:15.088517 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40829
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:15.431999 22362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:15.432338 22362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:15.432508 22362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:15.439934 22362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:15.440088 22362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:15.440316 22362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:50:15.451079 22362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:15.452473 22362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:15.453995 22362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:15.467494 22367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:15.467514 22368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:15.467672 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:15.468178 22362 server_base.cc:1034] running on GCE node
I20260619 04:50:15.468775 22362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:15.469643 22362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:15.470839 22362 hybrid_clock.cc:648] HybridClock initialized: now 1781844615470777 us; error 58 us; skew 500 ppm
I20260619 04:50:15.471153 22362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:15 dist-test-slave-2qp8 krb5kdc[22256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:15.479197 22362 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:50:15.480762 22362 webserver.cc:460] Webserver started at http://127.3.243.129:36867/ using document root <none> and password file <none>
I20260619 04:50:15.481485 22362 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:15.481626 22362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:15.481961 22362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:15.485050 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "ac24016588af4d38a34c3b232386e2f7"
format_stamp: "Formatted at 2026-06-19 04:50:15 on dist-test-slave-2qp8"
server_key: "da7552de8a561443a06ad4a5a6e3649d"
server_key_iv: "ce998c07ccf1415083109a2b44b96fe2"
server_key_version: "encryptionkey@0"
I20260619 04:50:15.485831 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "ac24016588af4d38a34c3b232386e2f7"
format_stamp: "Formatted at 2026-06-19 04:50:15 on dist-test-slave-2qp8"
server_key: "da7552de8a561443a06ad4a5a6e3649d"
server_key_iv: "ce998c07ccf1415083109a2b44b96fe2"
server_key_version: "encryptionkey@0"
I20260619 04:50:15.490760 22362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:50:15.494117 22377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:15.495312 22362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:50:15.495482 22362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "ac24016588af4d38a34c3b232386e2f7"
format_stamp: "Formatted at 2026-06-19 04:50:15 on dist-test-slave-2qp8"
server_key: "da7552de8a561443a06ad4a5a6e3649d"
server_key_iv: "ce998c07ccf1415083109a2b44b96fe2"
server_key_version: "encryptionkey@0"
I20260619 04:50:15.495672 22362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:15.517077 22362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:15.521536 22362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:15.521862 22362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:15.522786 22362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:15.524163 22362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:15.524241 22362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260619 04:50:15.524359 22362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:15.524405 22362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:15.557610 22362 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:37381
I20260619 04:50:15.557709 22489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:37381 every 8 connection(s)
I20260619 04:50:15.558894 22362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:50:15.567502 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 22362
I20260619 04:50:15.567730 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:50:15.568302 4046 external_mini_cluster.cc:1428] Setting key f05f78f4a07c3e698a40fe8f8cc94eb7
Jun 19 04:50:15 dist-test-slave-2qp8 krb5kdc[22256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:15.580276 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:15.561501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:51341 (local address 127.3.243.190:40829)
0619 04:50:15.561965 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:15.562003 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:15.563057 (+ 1054us) server_negotiation.cc:407] Connection header received
0619 04:50:15.565038 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:15.565042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:15.565111 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:15.565211 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:15.566666 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:15.567387 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:15.569637 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:15.570274 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:15.570411 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:15.574516 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:15.574521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:15.574524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:15.574554 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:15.577193 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:15.577934 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:15.577945 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:15.577947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:15.578059 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:15.578495 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:15.578500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:15.578503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:15.578783 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:15.578947 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:15.579637 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 04:50:15.579878 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":216,"threads_started":1}
I20260619 04:50:15.581257 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:15.562150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:40829 (local address 127.3.243.129:51341)
0619 04:50:15.562863 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:15.562919 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:15.564793 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:15.565430 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:15.565438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:15.565977 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:15.566518 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:15.566537 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:15.567567 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:15.567576 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:15.569496 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:15.569507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:15.570543 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:15.570551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:15.570837 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:15.571567 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:50:15.571600 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:15.574280 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:15.577349 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:15.577357 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:15.577372 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:15.577806 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:15.578170 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:15.578175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:15.578179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:15.578381 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:15.578918 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:15.578926 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:15.579262 (+ 336us) client_negotiation.cc:769] Sending connection context
0619 04:50:15.579532 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 04:50:15.580056 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":257,"threads_started":1}
I20260619 04:50:15.582863 22490 heartbeater.cc:347] Connected to a master server at 127.3.243.190:40829
I20260619 04:50:15.583195 22490 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:15.584043 22490 heartbeater.cc:511] Master 127.3.243.190:40829 requested a full tablet report, sending...
I20260619 04:50:15.586534 22301 ts_manager.cc:194] Registered new tserver with Master: ac24016588af4d38a34c3b232386e2f7 (127.3.243.129:37381)
I20260619 04:50:15.588714 22301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:51341
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:50:15.627082 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40829
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:15.964334 22496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:15.964599 22496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:15.964690 22496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:15.971694 22496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:15.971798 22496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:15.971998 22496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:50:15.981267 22496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:15.982515 22496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:15.983850 22496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:15.997366 22501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:15.997331 22504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:15.997394 22502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:15.998260 22496 server_base.cc:1034] running on GCE node
I20260619 04:50:15.999012 22496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:16.000020 22496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:16.001277 22496 hybrid_clock.cc:648] HybridClock initialized: now 1781844616001223 us; error 84 us; skew 500 ppm
I20260619 04:50:16.001632 22496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:16 dist-test-slave-2qp8 krb5kdc[22256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:16.004904 22496 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:50:16.006531 22496 webserver.cc:460] Webserver started at http://127.3.243.130:40851/ using document root <none> and password file <none>
I20260619 04:50:16.007257 22496 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:16.007411 22496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:16.007772 22496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:16.010841 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "891a2790f77943abb8ab7f10496ea9b9"
format_stamp: "Formatted at 2026-06-19 04:50:16 on dist-test-slave-2qp8"
server_key: "655b7933d72f3aefa58d200b781d1eb8"
server_key_iv: "f56707ac868cc65da1dadf7c8b9161fb"
server_key_version: "encryptionkey@0"
I20260619 04:50:16.011708 22496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "891a2790f77943abb8ab7f10496ea9b9"
format_stamp: "Formatted at 2026-06-19 04:50:16 on dist-test-slave-2qp8"
server_key: "655b7933d72f3aefa58d200b781d1eb8"
server_key_iv: "f56707ac868cc65da1dadf7c8b9161fb"
server_key_version: "encryptionkey@0"
I20260619 04:50:16.017096 22496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260619 04:50:16.020750 22511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:16.022125 22496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:50:16.022301 22496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "891a2790f77943abb8ab7f10496ea9b9"
format_stamp: "Formatted at 2026-06-19 04:50:16 on dist-test-slave-2qp8"
server_key: "655b7933d72f3aefa58d200b781d1eb8"
server_key_iv: "f56707ac868cc65da1dadf7c8b9161fb"
server_key_version: "encryptionkey@0"
I20260619 04:50:16.022501 22496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:16.038877 22496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:16.042776 22496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:16.043062 22496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:16.043995 22496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:16.045338 22496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:16.045439 22496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:16.045588 22496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:16.045663 22496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:16.078379 22496 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:44599
I20260619 04:50:16.078425 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:44599 every 8 connection(s)
I20260619 04:50:16.079641 22496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:50:16.083592 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 22496
I20260619 04:50:16.083786 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:50:16.084384 4046 external_mini_cluster.cc:1428] Setting key 4f715319fd0510c58fa70a2152373492
Jun 19 04:50:16 dist-test-slave-2qp8 krb5kdc[22256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:16.100615 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.082195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:48957 (local address 127.3.243.190:40829)
0619 04:50:16.082651 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.082655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.083834 (+ 1179us) server_negotiation.cc:407] Connection header received
0619 04:50:16.086333 (+ 2499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.086338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.086419 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.086531 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:16.088001 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.088764 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.090309 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.090741 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.090873 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.094298 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:16.094303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:16.094307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:16.094339 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:16.097432 (+ 3093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:16.098156 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:16.098163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:16.098171 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:16.098303 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:16.098720 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:16.098726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:16.098729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:16.099097 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:16.099246 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.099931 (+ 685us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.100214 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":228,"threads_started":1}
I20260619 04:50:16.101619 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.082828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:40829 (local address 127.3.243.130:48957)
0619 04:50:16.083645 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:16.083705 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:16.086013 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:16.086770 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:16.086778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:16.087293 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:16.087838 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.087855 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.088925 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.088935 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.090164 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.090175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.090956 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.090964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.091189 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.091855 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:50:16.091877 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:16.094081 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:16.097579 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:16.097590 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:16.097608 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:16.098033 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:16.098413 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:16.098419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:16.098423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:16.098610 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:16.099218 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:16.099225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:16.099586 (+ 361us) client_negotiation.cc:769] Sending connection context
0619 04:50:16.099851 (+ 265us) client_negotiation.cc:240] Negotiation successful
0619 04:50:16.100461 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":289,"threads_started":1}
I20260619 04:50:16.103166 22624 heartbeater.cc:347] Connected to a master server at 127.3.243.190:40829
I20260619 04:50:16.103482 22624 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:16.104431 22624 heartbeater.cc:511] Master 127.3.243.190:40829 requested a full tablet report, sending...
I20260619 04:50:16.105930 22301 ts_manager.cc:194] Registered new tserver with Master: 891a2790f77943abb8ab7f10496ea9b9 (127.3.243.130:44599)
I20260619 04:50:16.106734 22301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:48957
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:50:16.143275 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40829
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:44875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:16.475330 22630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:16.475593 22630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:16.475746 22630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:16.482755 22630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:16.482849 22630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:16.482954 22630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:50:16.492488 22630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:44875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40829
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:16.493666 22630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:16.494854 22630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:16.508443 22635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:16.508368 22636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:16.508417 22638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:16.509480 22630 server_base.cc:1034] running on GCE node
I20260619 04:50:16.510010 22630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:16.510859 22630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:16.512059 22630 hybrid_clock.cc:648] HybridClock initialized: now 1781844616512025 us; error 28 us; skew 500 ppm
I20260619 04:50:16.512374 22630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:16 dist-test-slave-2qp8 krb5kdc[22256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:16.515434 22630 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:50:16.516847 22630 webserver.cc:460] Webserver started at http://127.3.243.131:43783/ using document root <none> and password file <none>
I20260619 04:50:16.517514 22630 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:16.517642 22630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:16.517956 22630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:16.521036 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "1c129bb7cf9e4e0684b139febbfa3a30"
format_stamp: "Formatted at 2026-06-19 04:50:16 on dist-test-slave-2qp8"
server_key: "8fde9c4fcda1a33c4f15b6ac60e95259"
server_key_iv: "4d44ef5b4722d2513bc4d895e51eeed7"
server_key_version: "encryptionkey@0"
I20260619 04:50:16.521955 22630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "1c129bb7cf9e4e0684b139febbfa3a30"
format_stamp: "Formatted at 2026-06-19 04:50:16 on dist-test-slave-2qp8"
server_key: "8fde9c4fcda1a33c4f15b6ac60e95259"
server_key_iv: "4d44ef5b4722d2513bc4d895e51eeed7"
server_key_version: "encryptionkey@0"
I20260619 04:50:16.527060 22630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:50:16.530587 22645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:16.531879 22630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:50:16.532034 22630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "1c129bb7cf9e4e0684b139febbfa3a30"
format_stamp: "Formatted at 2026-06-19 04:50:16 on dist-test-slave-2qp8"
server_key: "8fde9c4fcda1a33c4f15b6ac60e95259"
server_key_iv: "4d44ef5b4722d2513bc4d895e51eeed7"
server_key_version: "encryptionkey@0"
I20260619 04:50:16.532274 22630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:16.553860 22630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:16.558126 22630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:16.558418 22630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:16.559396 22630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:16.560678 22630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:16.560752 22630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:16.560918 22630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:16.560988 22630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:16.592594 22490 heartbeater.cc:503] Master 127.3.243.190:40829 was elected leader, sending a full tablet report...
I20260619 04:50:16.594560 22630 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:44297
I20260619 04:50:16.594594 22757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:44297 every 8 connection(s)
I20260619 04:50:16.595877 22630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:50:16.598049 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 22630
I20260619 04:50:16.598213 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:50:16.598706 4046 external_mini_cluster.cc:1428] Setting key a5f4b665e78b8916653f9c864ac37873
Jun 19 04:50:16 dist-test-slave-2qp8 krb5kdc[22256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:16.615782 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.599462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:43979 (local address 127.3.243.190:40829)
0619 04:50:16.599705 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.599712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.599947 (+ 235us) server_negotiation.cc:407] Connection header received
0619 04:50:16.602524 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.602529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.602627 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.602747 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:16.604261 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.605019 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.606549 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.606966 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.607120 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.610390 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:16.610403 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:16.610406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:16.610438 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:16.612891 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:16.613609 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:16.613613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:16.613615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:16.613734 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:16.614149 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:16.614154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:16.614157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:16.614412 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:16.614578 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.615235 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.615460 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:50:16.616797 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.599047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:40829 (local address 127.3.243.131:43979)
0619 04:50:16.599803 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:16.599845 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:16.602232 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:16.602983 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:16.602991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:16.603508 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:16.604061 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.604081 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.605167 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.605172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.606411 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.606423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.607226 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.607234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.607475 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.608050 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:50:16.608071 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:16.610192 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:16.613035 (+ 2843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:16.613043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:16.613060 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:16.613474 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:16.613843 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:16.613851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:16.613856 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:16.614031 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:16.614538 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:16.614545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:16.614880 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 04:50:16.615134 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 04:50:16.615657 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":693,"threads_started":1}
I20260619 04:50:16.618059 22758 heartbeater.cc:347] Connected to a master server at 127.3.243.190:40829
I20260619 04:50:16.618364 22758 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:16.619065 22758 heartbeater.cc:511] Master 127.3.243.190:40829 requested a full tablet report, sending...
I20260619 04:50:16.620432 22301 ts_manager.cc:194] Registered new tserver with Master: 1c129bb7cf9e4e0684b139febbfa3a30 (127.3.243.131:44297)
I20260619 04:50:16.621222 22301 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:43979
I20260619 04:50:16.626138 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:50:16.643532 22626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.630156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42778 (local address 127.3.243.190:40829)
0619 04:50:16.630420 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.630429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.630737 (+ 308us) server_negotiation.cc:407] Connection header received
0619 04:50:16.630879 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.630882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.630940 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.631018 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:16.632426 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.633147 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.634409 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.634808 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.634895 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.638302 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:16.638307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:16.638310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:16.638347 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:16.640866 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:16.641649 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:16.641653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:16.641655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:16.641765 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:16.642161 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:16.642164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:16.642166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:16.642442 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:16.642641 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.643024 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.643144 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260619 04:50:16.647403 22301 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:50:16.654284 22301 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:50:16.691617 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.666785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:44297 (local address 127.0.0.1:36312)
0619 04:50:16.681941 (+ 15156us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:16.681969 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:16.682146 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:16.683989 (+ 1843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:16.683995 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:16.684023 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:16.684526 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.684541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.686358 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.686364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.688582 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.688597 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.690686 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.690690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.690843 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.690879 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:50:16.691013 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 04:50:16.691197 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14980,"mutex_wait_us":36,"thread_start_us":981,"threads_started":1}
I20260619 04:50:16.691588 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.667123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.3.243.131:44297)
0619 04:50:16.681458 (+ 14335us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.681470 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.682722 (+ 1252us) server_negotiation.cc:407] Connection header received
0619 04:50:16.682874 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.682882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.683179 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.683417 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:16.684741 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.686222 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.688779 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.690265 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.690381 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.690595 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.691071 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.691251 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14128,"thread_start_us":13971,"threads_started":1}
I20260619 04:50:16.692781 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.666526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:44599 (local address 127.0.0.1:41278)
0619 04:50:16.668195 (+ 1669us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:16.668255 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:16.668410 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:16.685096 (+ 16686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:16.685102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:16.685128 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:16.685576 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.685585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.687551 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.687556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.689939 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.689950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.691934 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.691943 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.692165 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.692193 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 04:50:16.692285 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 04:50:16.692409 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"mutex_wait_us":6,"thread_start_us":17405,"threads_started":1}
I20260619 04:50:16.692862 22770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.668637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37381 (local address 127.0.0.1:56782)
0619 04:50:16.683685 (+ 15048us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:16.683745 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:16.684254 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:16.685240 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:16.685245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:16.685275 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:16.685647 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.685658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.687672 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.687675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.690195 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.690211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.692154 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.692161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.692367 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.692400 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:50:16.692472 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 04:50:16.692556 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14899,"mutex_wait_us":154,"thread_start_us":15048,"threads_started":1}
I20260619 04:50:16.693631 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.666705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41278 (local address 127.3.243.130:44599)
0619 04:50:16.684579 (+ 17874us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.684588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.684618 (+ 30us) server_negotiation.cc:407] Connection header received
0619 04:50:16.684741 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.684747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.684913 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.685114 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:16.685754 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.687404 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.690086 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.691470 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.691607 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.691787 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.693159 (+ 1372us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.693326 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17712,"thread_start_us":17591,"threads_started":1}
I20260619 04:50:16.693730 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.668941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.3.243.129:37381)
0619 04:50:16.682757 (+ 13816us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.682765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.683881 (+ 1116us) server_negotiation.cc:407] Connection header received
0619 04:50:16.684465 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.684473 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.684740 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.684930 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:16.685875 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.687497 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.690379 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.691885 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.692027 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.693203 (+ 1176us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.693270 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.693435 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13638,"thread_start_us":13495,"threads_started":1}
I20260619 04:50:16.696197 22693 tablet_service.cc:1467] Processing CreateTablet for tablet 3934a2383f4343ce9bde4de15466318d (DEFAULT_TABLE table=test-table [id=e067ed59a6764835ae17f93c111c7935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:16.697805 22693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3934a2383f4343ce9bde4de15466318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:16.697355 22559 tablet_service.cc:1467] Processing CreateTablet for tablet 3934a2383f4343ce9bde4de15466318d (DEFAULT_TABLE table=test-table [id=e067ed59a6764835ae17f93c111c7935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:16.698274 22425 tablet_service.cc:1467] Processing CreateTablet for tablet 3934a2383f4343ce9bde4de15466318d (DEFAULT_TABLE table=test-table [id=e067ed59a6764835ae17f93c111c7935]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:16.698844 22559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3934a2383f4343ce9bde4de15466318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:16.699431 22425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3934a2383f4343ce9bde4de15466318d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:16.710541 22776 tablet_bootstrap.cc:492] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: Bootstrap starting.
I20260619 04:50:16.712723 22777 tablet_bootstrap.cc:492] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: Bootstrap starting.
I20260619 04:50:16.715261 22776 tablet_bootstrap.cc:654] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:16.716148 22777 tablet_bootstrap.cc:654] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:16.717356 22776 log.cc:824] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:16.717342 22777 log.cc:824] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:16.718963 22778 tablet_bootstrap.cc:492] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: Bootstrap starting.
I20260619 04:50:16.720717 22776 tablet_bootstrap.cc:492] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: No bootstrap required, opened a new log
I20260619 04:50:16.720710 22777 tablet_bootstrap.cc:492] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: No bootstrap required, opened a new log
I20260619 04:50:16.720986 22777 ts_tablet_manager.cc:1397] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 04:50:16.720986 22776 ts_tablet_manager.cc:1397] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 04:50:16.721830 22778 tablet_bootstrap.cc:654] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:16.722766 22778 log.cc:824] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:16.726243 22778 tablet_bootstrap.cc:492] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: No bootstrap required, opened a new log
I20260619 04:50:16.726496 22778 ts_tablet_manager.cc:1397] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260619 04:50:16.731791 22777 raft_consensus.cc:348] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } }
I20260619 04:50:16.732312 22777 raft_consensus.cc:374] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:16.732416 22777 raft_consensus.cc:729] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c129bb7cf9e4e0684b139febbfa3a30, State: Initialized, Role: FOLLOWER
I20260619 04:50:16.732215 22776 raft_consensus.cc:348] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } }
I20260619 04:50:16.732677 22776 raft_consensus.cc:374] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:16.732801 22776 raft_consensus.cc:729] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac24016588af4d38a34c3b232386e2f7, State: Initialized, Role: FOLLOWER
I20260619 04:50:16.733194 22777 consensus_queue.cc:260] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } }
I20260619 04:50:16.733335 22776 consensus_queue.cc:260] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } }
I20260619 04:50:16.734331 22758 heartbeater.cc:503] Master 127.3.243.190:40829 was elected leader, sending a full tablet report...
I20260619 04:50:16.735313 22777 ts_tablet_manager.cc:1428] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 04:50:16.737025 22776 ts_tablet_manager.cc:1428] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260619 04:50:16.737073 22778 raft_consensus.cc:348] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } }
I20260619 04:50:16.737516 22778 raft_consensus.cc:374] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:16.737612 22778 raft_consensus.cc:729] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891a2790f77943abb8ab7f10496ea9b9, State: Initialized, Role: FOLLOWER
I20260619 04:50:16.738200 22778 consensus_queue.cc:260] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } }
I20260619 04:50:16.739274 22624 heartbeater.cc:503] Master 127.3.243.190:40829 was elected leader, sending a full tablet report...
I20260619 04:50:16.739835 22778 ts_tablet_manager.cc:1428] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260619 04:50:16.806911 22782 raft_consensus.cc:482] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:50:16.807219 22782 raft_consensus.cc:504] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } }
I20260619 04:50:16.809007 22782 leader_election.cc:288] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 891a2790f77943abb8ab7f10496ea9b9 (127.3.243.130:44599), ac24016588af4d38a34c3b232386e2f7 (127.3.243.129:37381)
W20260619 04:50:16.811497 22491 tablet.cc:2369] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:16.815207 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.809528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:58281 (local address 127.3.243.129:37381)
0619 04:50:16.809710 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.809715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.809732 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:50:16.809793 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.809796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.809857 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.809962 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:16.810389 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.811299 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.813262 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.814464 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.814589 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.814638 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.814950 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.815037 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:50:16.815364 22760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.809262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37381 (local address 127.3.243.131:58281)
0619 04:50:16.809454 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:16.809474 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:16.809596 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:16.810006 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:16.810013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:16.810037 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:16.810224 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.810236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.811449 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.811457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.813105 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.813117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.814615 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.814622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.814795 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.814828 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:50:16.814923 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 04:50:16.815053 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":20}
I20260619 04:50:16.816305 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.809430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:45723 (local address 127.3.243.130:44599)
0619 04:50:16.809620 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.809627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.810529 (+ 902us) server_negotiation.cc:407] Connection header received
0619 04:50:16.810625 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.810632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.810709 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.810801 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:16.811354 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.812207 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.814302 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.815450 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.815610 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.815691 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.815947 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.816062 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:50:16.816435 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.809270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:44599 (local address 127.3.243.131:45723)
0619 04:50:16.810425 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:16.810438 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:16.810531 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:16.810837 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:16.810840 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:16.810860 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:16.811176 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.811187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.812359 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.812363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.814146 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:16.814161 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.815655 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.815665 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:16.815775 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.815796 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 04:50:16.815886 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 04:50:16.816031 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"mutex_wait_us":450,"thread_start_us":339,"threads_started":1}
I20260619 04:50:16.816468 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934a2383f4343ce9bde4de15466318d" candidate_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac24016588af4d38a34c3b232386e2f7" is_pre_election: true
I20260619 04:50:16.817057 22445 raft_consensus.cc:2454] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c129bb7cf9e4e0684b139febbfa3a30 in term 0.
I20260619 04:50:16.817416 22579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934a2383f4343ce9bde4de15466318d" candidate_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891a2790f77943abb8ab7f10496ea9b9" is_pre_election: true
I20260619 04:50:16.817691 22648 leader_election.cc:302] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c129bb7cf9e4e0684b139febbfa3a30, ac24016588af4d38a34c3b232386e2f7; no voters:
I20260619 04:50:16.817919 22579 raft_consensus.cc:2454] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c129bb7cf9e4e0684b139febbfa3a30 in term 0.
I20260619 04:50:16.818081 22782 raft_consensus.cc:2781] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:50:16.818231 22782 raft_consensus.cc:482] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:50:16.818362 22782 raft_consensus.cc:3037] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:16.820848 22782 raft_consensus.cc:504] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } }
I20260619 04:50:16.821448 22782 leader_election.cc:288] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [CANDIDATE]: Term 1 election: Requested vote from peers 891a2790f77943abb8ab7f10496ea9b9 (127.3.243.130:44599), ac24016588af4d38a34c3b232386e2f7 (127.3.243.129:37381)
I20260619 04:50:16.821857 22579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934a2383f4343ce9bde4de15466318d" candidate_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "891a2790f77943abb8ab7f10496ea9b9"
I20260619 04:50:16.821878 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3934a2383f4343ce9bde4de15466318d" candidate_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac24016588af4d38a34c3b232386e2f7"
I20260619 04:50:16.822068 22445 raft_consensus.cc:3037] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:16.822077 22579 raft_consensus.cc:3037] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:16.824252 22445 raft_consensus.cc:2454] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c129bb7cf9e4e0684b139febbfa3a30 in term 1.
I20260619 04:50:16.824612 22648 leader_election.cc:302] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c129bb7cf9e4e0684b139febbfa3a30, ac24016588af4d38a34c3b232386e2f7; no voters:
I20260619 04:50:16.824612 22579 raft_consensus.cc:2454] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c129bb7cf9e4e0684b139febbfa3a30 in term 1.
I20260619 04:50:16.824895 22782 raft_consensus.cc:2781] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:16.825380 22782 raft_consensus.cc:686] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 1 LEADER]: Becoming Leader. State: Replica: 1c129bb7cf9e4e0684b139febbfa3a30, State: Running, Role: LEADER
I20260619 04:50:16.826014 22782 consensus_queue.cc:237] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } }
I20260619 04:50:16.831066 22300 catalog_manager.cc:5495] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c129bb7cf9e4e0684b139febbfa3a30 (127.3.243.131). New cstate: current_term: 1 leader_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c129bb7cf9e4e0684b139febbfa3a30" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 44297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 } health_report { overall_health: UNKNOWN } } }
W20260619 04:50:16.831632 22625 tablet.cc:2369] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:50:16.847549 22759 tablet.cc:2369] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:16.869158 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.863553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.3.243.131:44297)
0619 04:50:16.863760 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.863765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.863785 (+ 20us) server_negotiation.cc:407] Connection header received
0619 04:50:16.864010 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.864014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.864101 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.864309 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:50:16.864704 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.865475 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.866846 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.867245 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.867379 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.867991 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:50:16.868543 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:50:16.868725 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.868883 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.868941 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:50:16.877120 22579 raft_consensus.cc:1261] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [term 1 FOLLOWER]: Refusing update from remote peer 1c129bb7cf9e4e0684b139febbfa3a30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:16.877108 22445 raft_consensus.cc:1261] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [term 1 FOLLOWER]: Refusing update from remote peer 1c129bb7cf9e4e0684b139febbfa3a30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:16.878027 22786 consensus_queue.cc:1035] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac24016588af4d38a34c3b232386e2f7" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:16.878415 22782 consensus_queue.cc:1035] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "891a2790f77943abb8ab7f10496ea9b9" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:16.906080 22301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:16.906386 22301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:16.910048 22301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1cf32a430f24a779e6239b42f72bdb3: Sending DeleteTablet for 3 replicas of tablet 3934a2383f4343ce9bde4de15466318d
I20260619 04:50:16.911144 22559 tablet_service.cc:1514] Processing DeleteTablet for tablet 3934a2383f4343ce9bde4de15466318d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:16 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:41278
I20260619 04:50:16.911267 22693 tablet_service.cc:1514] Processing DeleteTablet for tablet 3934a2383f4343ce9bde4de15466318d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:16 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:36312
I20260619 04:50:16.912266 22425 tablet_service.cc:1514] Processing DeleteTablet for tablet 3934a2383f4343ce9bde4de15466318d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:16 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:56782
I20260619 04:50:16.912415 22799 tablet_replica.cc:330] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: stopping tablet replica
I20260619 04:50:16.912577 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 22362
I20260619 04:50:16.912822 22799 raft_consensus.cc:2229] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:16.913419 22799 raft_consensus.cc:2258] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:16.914258 22800 tablet_replica.cc:330] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: stopping tablet replica
I20260619 04:50:16.914721 22800 raft_consensus.cc:2229] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:50:16.915511 22800 raft_consensus.cc:2258] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:16.918182 22799 ts_tablet_manager.cc:1905] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:16.923995 22801 tablet_replica.cc:330] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: stopping tablet replica
I20260619 04:50:16.924548 22801 raft_consensus.cc:2229] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:16.925127 22801 raft_consensus.cc:2258] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:16.926304 22799 ts_tablet_manager.cc:1918] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:16.926527 22799 log.cc:1196] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/3934a2383f4343ce9bde4de15466318d
I20260619 04:50:16.927062 22799 ts_tablet_manager.cc:1939] T 3934a2383f4343ce9bde4de15466318d P 891a2790f77943abb8ab7f10496ea9b9: Deleting consensus metadata
I20260619 04:50:16.927464 22800 ts_tablet_manager.cc:1905] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:16.928897 22286 catalog_manager.cc:4841] TS 891a2790f77943abb8ab7f10496ea9b9 (127.3.243.130:44599): tablet 3934a2383f4343ce9bde4de15466318d (table test-table [id=e067ed59a6764835ae17f93c111c7935]) successfully deleted
I20260619 04:50:16.932638 22801 ts_tablet_manager.cc:1905] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 04:50:16 dist-test-slave-2qp8 krb5kdc[22256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:50:16.941447 22800 ts_tablet_manager.cc:1918] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:16.941627 22800 log.cc:1196] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/3934a2383f4343ce9bde4de15466318d
I20260619 04:50:16.942173 22800 ts_tablet_manager.cc:1939] T 3934a2383f4343ce9bde4de15466318d P 1c129bb7cf9e4e0684b139febbfa3a30: Deleting consensus metadata
I20260619 04:50:16.944021 22287 catalog_manager.cc:4841] TS 1c129bb7cf9e4e0684b139febbfa3a30 (127.3.243.131:44297): tablet 3934a2383f4343ce9bde4de15466318d (table test-table [id=e067ed59a6764835ae17f93c111c7935]) successfully deleted
I20260619 04:50:16.945336 22801 ts_tablet_manager.cc:1918] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:16.945519 22801 log.cc:1196] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/3934a2383f4343ce9bde4de15466318d
I20260619 04:50:16.946108 22801 ts_tablet_manager.cc:1939] T 3934a2383f4343ce9bde4de15466318d P ac24016588af4d38a34c3b232386e2f7: Deleting consensus metadata
I20260619 04:50:16.946404 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:16.915253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.3.243.129:37381)
0619 04:50:16.915425 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:16.915431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:16.921110 (+ 5679us) server_negotiation.cc:407] Connection header received
0619 04:50:16.921306 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:16.921312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:16.921414 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:16.921514 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:16.929398 (+ 7884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.930449 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.932339 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:16.932945 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:16.933041 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:16.938865 (+ 5824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:16.938879 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:16.938901 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:16.938949 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:16.942938 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:16.943835 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:16.943844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:16.943848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:16.944024 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:16.944585 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:16.944591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:16.944594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:16.945087 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:16.945419 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:16.945734 (+ 315us) server_negotiation.cc:299] Negotiation successful
0619 04:50:16.945958 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:50:16.947057 22485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:17.037060 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 22362
I20260619 04:50:17.037277 22287 catalog_manager.cc:4841] TS ac24016588af4d38a34c3b232386e2f7 (127.3.243.129:37381): tablet 3934a2383f4343ce9bde4de15466318d (table test-table [id=e067ed59a6764835ae17f93c111c7935]) successfully deleted
I20260619 04:50:17.055833 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 22496
Jun 19 04:50:17 dist-test-slave-2qp8 krb5kdc[22256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:50:17.072980 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:17.056774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.3.243.130:44599)
0619 04:50:17.056974 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:17.056980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:17.057003 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:50:17.057073 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:17.057079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:17.057164 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:17.057275 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:17.058467 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:17.059608 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:17.061112 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:17.061794 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:17.061930 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:17.066264 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:17.066283 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:17.066304 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:17.066349 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:17.070044 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:17.070781 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:17.070793 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:17.070797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:17.070961 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:17.071290 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:17.071297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:17.071301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:17.071781 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:17.072093 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:17.072369 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 04:50:17.072596 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:50:17.073756 22619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:17.163462 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 22496
I20260619 04:50:17.180790 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 22630
Jun 19 04:50:17 dist-test-slave-2qp8 krb5kdc[22256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:50:17.197160 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:17.181627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.3.243.131:44297)
0619 04:50:17.181820 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:17.181827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:17.181953 (+ 126us) server_negotiation.cc:407] Connection header received
0619 04:50:17.182176 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:17.182182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:17.182266 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:17.182390 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:17.183635 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:17.184589 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:17.185820 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:17.186282 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:17.186438 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:17.190831 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:17.190844 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:17.190879 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:17.190917 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:17.194269 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:17.194913 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:17.194926 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:17.194928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:17.195099 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:17.195500 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:17.195507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:17.195515 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:17.195861 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:17.196290 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:17.196578 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 04:50:17.196797 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 04:50:17.197813 22753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:17.318558 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 22630
I20260619 04:50:17.337925 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 22269
I20260619 04:50:17.338974 22331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:17.449195 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 22269
2026-06-19T04:50:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4363 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T04:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:17Z Disabled control of system clock
I20260619 04:50:17.525815 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:35135
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:35135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:50:17.852111 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:17.852417 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:17.852497 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:17.859557 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:17.859660 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:17.859696 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:17.859767 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:17.859812 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:17.868803 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:35135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:35135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:17.869989 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:17.871682 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:17.883531 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:17.883559 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:17.883507 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:17.884466 22810 server_base.cc:1034] running on GCE node
I20260619 04:50:17.885236 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:17.886713 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:17.887907 22810 hybrid_clock.cc:648] HybridClock initialized: now 1781844617887849 us; error 65 us; skew 500 ppm
I20260619 04:50:17.888284 22810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:17.890725 22810 webserver.cc:460] Webserver started at http://127.3.243.190:35331/ using document root <none> and password file <none>
I20260619 04:50:17.891376 22810 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:17.891464 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:17.891716 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:17.894585 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "40cfa469754a49bfb5ecf5a9a7428eaf"
format_stamp: "Formatted at 2026-06-19 04:50:17 on dist-test-slave-2qp8"
server_key: "66a04cad79772f63a448a616dff15f64"
server_key_iv: "8d4193652b3315d7bb07f1ac938a1188"
server_key_version: "encryptionkey@0"
I20260619 04:50:17.895372 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "40cfa469754a49bfb5ecf5a9a7428eaf"
format_stamp: "Formatted at 2026-06-19 04:50:17 on dist-test-slave-2qp8"
server_key: "66a04cad79772f63a448a616dff15f64"
server_key_iv: "8d4193652b3315d7bb07f1ac938a1188"
server_key_version: "encryptionkey@0"
I20260619 04:50:17.900444 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:50:17.903738 22824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:17.905100 22810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:17.905233 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "40cfa469754a49bfb5ecf5a9a7428eaf"
format_stamp: "Formatted at 2026-06-19 04:50:17 on dist-test-slave-2qp8"
server_key: "66a04cad79772f63a448a616dff15f64"
server_key_iv: "8d4193652b3315d7bb07f1ac938a1188"
server_key_version: "encryptionkey@0"
I20260619 04:50:17.905409 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:17.920282 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:17.921070 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:17.921301 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:17.940201 22810 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:35135
I20260619 04:50:17.940244 22875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:35135 every 8 connection(s)
I20260619 04:50:17.941625 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:17.945547 22876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:17.949203 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 22810
I20260619 04:50:17.949388 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:17.949939 4046 external_mini_cluster.cc:1428] Setting key 4c8a6687535d05498e628c3cf5db754e
I20260619 04:50:17.958132 22876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf: Bootstrap starting.
I20260619 04:50:17.961303 22876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:17.962457 22876 log.cc:824] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:17.963385 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:17.951932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.3.243.190:35135)
0619 04:50:17.952944 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:17.952956 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:17.952995 (+ 39us) server_negotiation.cc:407] Connection header received
0619 04:50:17.954630 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:17.954653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:17.955127 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:17.955535 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:17.956173 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:17.957405 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:17.958772 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:17.959461 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:17.959669 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:17.960783 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:17.960790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:17.960804 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:17.961373 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:17.961507 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:17.961541 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:17.961663 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:17.961893 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 04:50:17.962201 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":325,"threads_started":1}
I20260619 04:50:17.965592 22876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf: No bootstrap required, opened a new log
I20260619 04:50:17.973166 22876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cfa469754a49bfb5ecf5a9a7428eaf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35135 } }
I20260619 04:50:17.973546 22876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:17.973649 22876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40cfa469754a49bfb5ecf5a9a7428eaf, State: Initialized, Role: FOLLOWER
I20260619 04:50:17.974179 22876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cfa469754a49bfb5ecf5a9a7428eaf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35135 } }
I20260619 04:50:17.974426 22876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:17.974565 22876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:17.974726 22876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:17.977053 22876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cfa469754a49bfb5ecf5a9a7428eaf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35135 } }
I20260619 04:50:17.977496 22876 leader_election.cc:302] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40cfa469754a49bfb5ecf5a9a7428eaf; no voters:
I20260619 04:50:17.978005 22876 leader_election.cc:288] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:17.978149 22881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:17.978875 22881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [term 1 LEADER]: Becoming Leader. State: Replica: 40cfa469754a49bfb5ecf5a9a7428eaf, State: Running, Role: LEADER
I20260619 04:50:17.979347 22881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cfa469754a49bfb5ecf5a9a7428eaf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35135 } }
I20260619 04:50:17.980147 22876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:17.982227 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40cfa469754a49bfb5ecf5a9a7428eaf. Latest consensus state: current_term: 1 leader_uuid: "40cfa469754a49bfb5ecf5a9a7428eaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cfa469754a49bfb5ecf5a9a7428eaf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35135 } } }
I20260619 04:50:17.982689 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:17.983075 22882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40cfa469754a49bfb5ecf5a9a7428eaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40cfa469754a49bfb5ecf5a9a7428eaf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 35135 } } }
I20260619 04:50:17.983430 22882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:17.984320 22887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:17.989856 22887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:17.998373 22887 catalog_manager.cc:1347] Generated new cluster ID: c3cbc7d72ffb4ec1a759453cb11061bd
I20260619 04:50:17.998517 22887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:18.021704 22887 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:18.022989 22887 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:18.036577 22887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf: Generated new TSK 0
I20260619 04:50:18.037508 22887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:18.052793 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:35135
--builtin_ntp_servers=127.3.243.148:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:50:18.405824 22900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:18.406113 22900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:18.406267 22900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:18.413797 22900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:18.413897 22900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:18.414080 22900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:50:18.423686 22900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:35135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:18.425185 22900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:18.426951 22900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:18.440320 22906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:18.440505 22905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:18.440438 22908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:18.441296 22900 server_base.cc:1034] running on GCE node
I20260619 04:50:18.441877 22900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:18.442776 22900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:18.444032 22900 hybrid_clock.cc:648] HybridClock initialized: now 1781844618443995 us; error 51 us; skew 500 ppm
I20260619 04:50:18.444399 22900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:18.446612 22900 webserver.cc:460] Webserver started at http://127.3.243.129:45195/ using document root <none> and password file <none>
I20260619 04:50:18.447324 22900 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:18.447440 22900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:18.447765 22900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:18.450824 22900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "a0d3a6d88a8e4502844faf0dc7547f0b"
format_stamp: "Formatted at 2026-06-19 04:50:18 on dist-test-slave-2qp8"
server_key: "297b07bd283c2e46f162c767ef08a080"
server_key_iv: "67042162d76da8e8b5c84f5625140108"
server_key_version: "encryptionkey@0"
I20260619 04:50:18.451643 22900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "a0d3a6d88a8e4502844faf0dc7547f0b"
format_stamp: "Formatted at 2026-06-19 04:50:18 on dist-test-slave-2qp8"
server_key: "297b07bd283c2e46f162c767ef08a080"
server_key_iv: "67042162d76da8e8b5c84f5625140108"
server_key_version: "encryptionkey@0"
I20260619 04:50:18.456878 22900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:50:18.460193 22914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:18.461491 22900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:50:18.461666 22900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "a0d3a6d88a8e4502844faf0dc7547f0b"
format_stamp: "Formatted at 2026-06-19 04:50:18 on dist-test-slave-2qp8"
server_key: "297b07bd283c2e46f162c767ef08a080"
server_key_iv: "67042162d76da8e8b5c84f5625140108"
server_key_version: "encryptionkey@0"
I20260619 04:50:18.461925 22900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:18.486469 22900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:18.487315 22900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:18.487617 22900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:18.488610 22900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:18.490036 22900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:18.490154 22900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:18.490299 22900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:18.490396 22900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:18.523011 22900 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:37243
I20260619 04:50:18.523100 23026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:37243 every 8 connection(s)
I20260619 04:50:18.524407 22900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:50:18.531898 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 22900
I20260619 04:50:18.532064 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:50:18.532508 4046 external_mini_cluster.cc:1428] Setting key 03512d970216046cdb48ed4dc5228aaa
I20260619 04:50:18.535506 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:35135
--builtin_ntp_servers=127.3.243.148:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:50:18.537315 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:18.526947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:53321 (local address 127.3.243.190:35135)
0619 04:50:18.527378 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:18.527383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:18.528603 (+ 1220us) server_negotiation.cc:407] Connection header received
0619 04:50:18.530368 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:18.530372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:18.530438 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:18.530534 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:18.531589 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:18.532423 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:18.534211 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:18.534772 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:18.534919 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:18.536264 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:18.536269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:18.536272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:18.536289 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:18.536331 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:18.536343 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:18.536437 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:18.536755 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 04:50:18.536868 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":213,"threads_started":1}
I20260619 04:50:18.538142 23030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:18.527532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35135 (local address 127.3.243.129:53321)
0619 04:50:18.528458 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:18.528493 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:18.530102 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:18.530731 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:18.530739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:18.530830 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:18.531440 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:18.531457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:18.532583 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:18.532587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:18.534023 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:18.534035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:18.535000 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:18.535005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:18.535203 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:18.535924 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:50:18.535927 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:18.535965 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:50:18.536043 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:18.536451 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:18.536464 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:18.536478 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 04:50:18.536693 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 04:50:18.537040 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":234,"threads_started":1}
I20260619 04:50:18.539734 23027 heartbeater.cc:347] Connected to a master server at 127.3.243.190:35135
I20260619 04:50:18.540053 23027 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:18.540776 23027 heartbeater.cc:511] Master 127.3.243.190:35135 requested a full tablet report, sending...
I20260619 04:50:18.542812 22841 ts_manager.cc:194] Registered new tserver with Master: a0d3a6d88a8e4502844faf0dc7547f0b (127.3.243.129:37243)
I20260619 04:50:18.544312 22841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.129:53321
W20260619 04:50:18.857422 23031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:18.857693 23031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:18.857833 23031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:18.865116 23031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:18.865240 23031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:18.865397 23031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:50:18.874351 23031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:35135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:18.875629 23031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:18.877381 23031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:18.890661 23039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:18.890846 23036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:18.890859 23037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:18.891948 23031 server_base.cc:1034] running on GCE node
I20260619 04:50:18.892627 23031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:18.893643 23031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:18.894946 23031 hybrid_clock.cc:648] HybridClock initialized: now 1781844618894875 us; error 55 us; skew 500 ppm
I20260619 04:50:18.895299 23031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:18.897537 23031 webserver.cc:460] Webserver started at http://127.3.243.130:46151/ using document root <none> and password file <none>
I20260619 04:50:18.898236 23031 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:18.898361 23031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:18.898675 23031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:18.901530 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "b8be9b48c8c14cbf9d5cfe28161705dc"
format_stamp: "Formatted at 2026-06-19 04:50:18 on dist-test-slave-2qp8"
server_key: "89acebd0a682265f75ba618ebc8d8a06"
server_key_iv: "73b0c46f6741622a64bf13bab33af815"
server_key_version: "encryptionkey@0"
I20260619 04:50:18.902446 23031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "b8be9b48c8c14cbf9d5cfe28161705dc"
format_stamp: "Formatted at 2026-06-19 04:50:18 on dist-test-slave-2qp8"
server_key: "89acebd0a682265f75ba618ebc8d8a06"
server_key_iv: "73b0c46f6741622a64bf13bab33af815"
server_key_version: "encryptionkey@0"
I20260619 04:50:18.907397 23031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:50:18.910861 23045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:18.912164 23031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:50:18.912317 23031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "b8be9b48c8c14cbf9d5cfe28161705dc"
format_stamp: "Formatted at 2026-06-19 04:50:18 on dist-test-slave-2qp8"
server_key: "89acebd0a682265f75ba618ebc8d8a06"
server_key_iv: "73b0c46f6741622a64bf13bab33af815"
server_key_version: "encryptionkey@0"
I20260619 04:50:18.912519 23031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:18.927783 23031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:18.928748 23031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:18.929056 23031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:18.929956 23031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:18.931309 23031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:18.931409 23031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:18.931564 23031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:18.931650 23031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:18.970057 23031 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:44043
I20260619 04:50:18.970100 23157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:44043 every 8 connection(s)
I20260619 04:50:18.971325 23031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:50:18.981207 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 23031
I20260619 04:50:18.981385 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:50:18.981884 4046 external_mini_cluster.cc:1428] Setting key a386c1fa8ca80c755f904ba496a7a02c
I20260619 04:50:18.983263 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:18.973861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:51595 (local address 127.3.243.190:35135)
0619 04:50:18.974028 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:18.974032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:18.975350 (+ 1318us) server_negotiation.cc:407] Connection header received
0619 04:50:18.977110 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:18.977114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:18.977179 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:18.977265 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:18.978209 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:18.978886 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:18.980399 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:18.980902 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:18.981023 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:18.982525 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:18.982530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:18.982533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:18.982550 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:18.982583 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:18.982595 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:18.982730 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:18.982949 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 04:50:18.983035 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:50:18.984699 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:18.974330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35135 (local address 127.3.243.130:51595)
0619 04:50:18.975191 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:18.975237 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:18.976885 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:18.977454 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:18.977462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:18.977540 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:18.978079 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:18.978093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:18.979002 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:18.979005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:18.980289 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:18.980299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:18.981127 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:18.981133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:18.981415 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:18.982186 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:50:18.982190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:18.982241 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:50:18.982316 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:18.982743 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:18.982752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:18.982758 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:50:18.982910 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 04:50:18.983306 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":253,"threads_started":1}
I20260619 04:50:18.985203 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:35135
--builtin_ntp_servers=127.3.243.148:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:50:18.986198 23158 heartbeater.cc:347] Connected to a master server at 127.3.243.190:35135
I20260619 04:50:18.986516 23158 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:18.987332 23158 heartbeater.cc:511] Master 127.3.243.190:35135 requested a full tablet report, sending...
I20260619 04:50:18.988829 22841 ts_manager.cc:194] Registered new tserver with Master: b8be9b48c8c14cbf9d5cfe28161705dc (127.3.243.130:44043)
I20260619 04:50:18.989681 22841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.130:51595
W20260619 04:50:19.302774 23161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:19.303087 23161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:19.303241 23161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:19.310424 23161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:19.310545 23161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:19.310709 23161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:50:19.319561 23161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:35135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:19.320873 23161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:19.322734 23161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:19.335830 23169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:19.335887 23166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:19.335860 23167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:19.336609 23161 server_base.cc:1034] running on GCE node
I20260619 04:50:19.337169 23161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:19.338043 23161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:19.339231 23161 hybrid_clock.cc:648] HybridClock initialized: now 1781844619339194 us; error 30 us; skew 500 ppm
I20260619 04:50:19.339540 23161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:19.341809 23161 webserver.cc:460] Webserver started at http://127.3.243.131:45147/ using document root <none> and password file <none>
I20260619 04:50:19.342461 23161 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:19.342541 23161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:19.342880 23161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:19.345726 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "1a65fbaa5b484a398e354d86d0f1ea2f"
format_stamp: "Formatted at 2026-06-19 04:50:19 on dist-test-slave-2qp8"
server_key: "fefdf91a0141dfd5b7d8b1f11a871eae"
server_key_iv: "4a07ed6a597fab624b0250b2dc2898aa"
server_key_version: "encryptionkey@0"
I20260619 04:50:19.346496 23161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "1a65fbaa5b484a398e354d86d0f1ea2f"
format_stamp: "Formatted at 2026-06-19 04:50:19 on dist-test-slave-2qp8"
server_key: "fefdf91a0141dfd5b7d8b1f11a871eae"
server_key_iv: "4a07ed6a597fab624b0250b2dc2898aa"
server_key_version: "encryptionkey@0"
I20260619 04:50:19.351277 23161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 04:50:19.354748 23175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:19.356037 23161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:50:19.356241 23161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "1a65fbaa5b484a398e354d86d0f1ea2f"
format_stamp: "Formatted at 2026-06-19 04:50:19 on dist-test-slave-2qp8"
server_key: "fefdf91a0141dfd5b7d8b1f11a871eae"
server_key_iv: "4a07ed6a597fab624b0250b2dc2898aa"
server_key_version: "encryptionkey@0"
I20260619 04:50:19.356467 23161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:19.378504 23161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:19.379329 23161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:19.379616 23161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:19.380609 23161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:19.382089 23161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:19.382208 23161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:19.382309 23161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:19.382404 23161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:19.415529 23161 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:36121
I20260619 04:50:19.415668 23287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:36121 every 8 connection(s)
I20260619 04:50:19.417065 23161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:50:19.419448 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 23161
I20260619 04:50:19.419636 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:50:19.420153 4046 external_mini_cluster.cc:1428] Setting key d4d7d3302b6bf5ff9df29bdb30ad3484
I20260619 04:50:19.429188 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.419512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:48873 (local address 127.3.243.190:35135)
0619 04:50:19.419741 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.419747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.421367 (+ 1620us) server_negotiation.cc:407] Connection header received
0619 04:50:19.423482 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.423486 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.423549 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.423631 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:19.424699 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.425413 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.426868 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.427254 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.427343 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.428569 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:19.428573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:19.428577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:19.428593 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:19.428622 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:19.428634 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:19.428701 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.428899 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.428967 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 04:50:19.430047 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.420096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:35135 (local address 127.3.243.131:48873)
0619 04:50:19.421214 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:19.421262 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:19.423239 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:19.423824 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:19.423831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:19.423936 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:19.424560 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.424577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.425538 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.425542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.426750 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.426762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.427421 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.427425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.427623 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.428290 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:50:19.428293 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:19.428329 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:50:19.428403 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:19.428714 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:19.428720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:19.428725 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 04:50:19.428859 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 04:50:19.429112 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":329,"threads_started":1}
I20260619 04:50:19.431205 23288 heartbeater.cc:347] Connected to a master server at 127.3.243.190:35135
I20260619 04:50:19.431493 23288 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:19.432245 23288 heartbeater.cc:511] Master 127.3.243.190:35135 requested a full tablet report, sending...
I20260619 04:50:19.433575 22841 ts_manager.cc:194] Registered new tserver with Master: 1a65fbaa5b484a398e354d86d0f1ea2f (127.3.243.131:36121)
I20260619 04:50:19.434592 22841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.131:48873
I20260619 04:50:19.435231 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:50:19.445348 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.439590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45780 (local address 127.3.243.190:35135)
0619 04:50:19.439771 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.439777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.440243 (+ 466us) server_negotiation.cc:407] Connection header received
0619 04:50:19.440492 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.440495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.440563 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.440639 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:19.441051 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.441703 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.443079 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.443477 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.443560 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.444655 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:19.444659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:19.444662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:19.444677 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:19.444706 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:19.444717 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:19.444833 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.444982 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.445048 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:50:19.449573 22841 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:50:19.456176 22841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:50:19.482719 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.468292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37243 (local address 127.0.0.1:45220)
0619 04:50:19.470417 (+ 2125us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:19.470476 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:19.470688 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:19.471565 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:19.471576 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:19.471616 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:19.472168 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.472185 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.475562 (+ 3377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.475571 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.478687 (+ 3116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.478708 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.481353 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.481363 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.481736 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.481818 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 04:50:19.481948 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 04:50:19.482196 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1783,"mutex_wait_us":4,"thread_start_us":786,"threads_started":1}
I20260619 04:50:19.484653 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.468472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45220 (local address 127.3.243.129:37243)
0619 04:50:19.469060 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.469071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.470624 (+ 1553us) server_negotiation.cc:407] Connection header received
0619 04:50:19.470941 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.470953 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.471265 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.472419 (+ 1154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:19.473365 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.475358 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.478998 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.481052 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.483694 (+ 2642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.484038 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.484125 (+ 87us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.484293 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":316,"threads_started":1}
I20260619 04:50:19.489084 22962 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc43894222f470a8b023e7f61872faf (DEFAULT_TABLE table=test-table [id=e67ca6d534df4de18442e86fa8fd8537]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:19.490602 22962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc43894222f470a8b023e7f61872faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:19.492758 23301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.468393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:36121 (local address 127.0.0.1:59898)
0619 04:50:19.471250 (+ 2857us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:19.471272 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:19.471365 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:19.485955 (+ 14590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:19.485962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:19.485992 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:19.486356 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.486369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.488242 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.488246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.490241 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.490255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.491938 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.491944 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.492149 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.492183 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:50:19.492299 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 04:50:19.492427 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2740,"thread_start_us":2630,"threads_started":1}
I20260619 04:50:19.492894 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.469001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.3.243.131:36121)
0619 04:50:19.484552 (+ 15551us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.484561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.484597 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:50:19.484701 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.484710 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.485003 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.485256 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:19.486583 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.488068 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.490456 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.491750 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.491871 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.492141 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.492367 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.492538 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15391,"thread_start_us":15089,"threads_started":1}
I20260619 04:50:19.496189 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.471394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:44043 (local address 127.0.0.1:60398)
0619 04:50:19.483378 (+ 11984us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:19.483401 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:19.483515 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:19.488578 (+ 5063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:19.488585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:19.488619 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:19.488972 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.488986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.491707 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.491711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.494128 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.494140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.495394 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.495400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.495569 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.495616 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 04:50:19.495715 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 04:50:19.495865 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11839,"mutex_wait_us":38,"thread_start_us":20090,"threads_started":1}
I20260619 04:50:19.496268 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.471958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.3.243.130:44043)
0619 04:50:19.487844 (+ 15886us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.487856 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.487891 (+ 35us) server_negotiation.cc:407] Connection header received
0619 04:50:19.488004 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.488014 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.488330 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.489148 (+ 818us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:19.489413 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.491212 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.494296 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.495173 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.495388 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.495628 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.495777 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.495931 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15685,"thread_start_us":19657,"threads_started":1}
I20260619 04:50:19.499801 23223 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc43894222f470a8b023e7f61872faf (DEFAULT_TABLE table=test-table [id=e67ca6d534df4de18442e86fa8fd8537]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:19.501202 23223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc43894222f470a8b023e7f61872faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:19.501152 23093 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc43894222f470a8b023e7f61872faf (DEFAULT_TABLE table=test-table [id=e67ca6d534df4de18442e86fa8fd8537]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:19.502375 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc43894222f470a8b023e7f61872faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:19.504732 23306 tablet_bootstrap.cc:492] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: Bootstrap starting.
I20260619 04:50:19.508572 23306 tablet_bootstrap.cc:654] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:19.509883 23306 log.cc:824] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:19.513074 23306 tablet_bootstrap.cc:492] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: No bootstrap required, opened a new log
I20260619 04:50:19.513353 23306 ts_tablet_manager.cc:1397] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 04:50:19.514397 23308 tablet_bootstrap.cc:492] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: Bootstrap starting.
I20260619 04:50:19.516851 23309 tablet_bootstrap.cc:492] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: Bootstrap starting.
I20260619 04:50:19.517704 23308 tablet_bootstrap.cc:654] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:19.518945 23308 log.cc:824] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:19.520579 23309 tablet_bootstrap.cc:654] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:19.522059 23309 log.cc:824] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:19.523463 23306 raft_consensus.cc:348] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } }
I20260619 04:50:19.523874 23306 raft_consensus.cc:374] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:19.523998 23306 raft_consensus.cc:729] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d3a6d88a8e4502844faf0dc7547f0b, State: Initialized, Role: FOLLOWER
I20260619 04:50:19.524677 23306 consensus_queue.cc:260] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } }
I20260619 04:50:19.527015 23306 ts_tablet_manager.cc:1428] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260619 04:50:19.527654 23027 heartbeater.cc:503] Master 127.3.243.190:35135 was elected leader, sending a full tablet report...
I20260619 04:50:19.529906 23309 tablet_bootstrap.cc:492] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: No bootstrap required, opened a new log
I20260619 04:50:19.530200 23309 ts_tablet_manager.cc:1397] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260619 04:50:19.532339 23308 tablet_bootstrap.cc:492] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: No bootstrap required, opened a new log
I20260619 04:50:19.532644 23308 ts_tablet_manager.cc:1397] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20260619 04:50:19.540277 23309 raft_consensus.cc:348] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } }
I20260619 04:50:19.540647 23309 raft_consensus.cc:374] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:19.540748 23309 raft_consensus.cc:729] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a65fbaa5b484a398e354d86d0f1ea2f, State: Initialized, Role: FOLLOWER
I20260619 04:50:19.540655 23308 raft_consensus.cc:348] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } }
I20260619 04:50:19.541091 23308 raft_consensus.cc:374] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:19.541216 23308 raft_consensus.cc:729] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8be9b48c8c14cbf9d5cfe28161705dc, State: Initialized, Role: FOLLOWER
I20260619 04:50:19.541396 23309 consensus_queue.cc:260] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } }
I20260619 04:50:19.541745 23308 consensus_queue.cc:260] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } }
I20260619 04:50:19.542448 23288 heartbeater.cc:503] Master 127.3.243.190:35135 was elected leader, sending a full tablet report...
I20260619 04:50:19.543134 23309 ts_tablet_manager.cc:1428] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 04:50:19.543954 23158 heartbeater.cc:503] Master 127.3.243.190:35135 was elected leader, sending a full tablet report...
I20260619 04:50:19.544608 23308 ts_tablet_manager.cc:1428] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
W20260619 04:50:19.668674 23289 tablet.cc:2369] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:19.672290 23313 raft_consensus.cc:482] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:50:19.672596 23313 raft_consensus.cc:504] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } }
I20260619 04:50:19.674127 23313 leader_election.cc:288] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d3a6d88a8e4502844faf0dc7547f0b (127.3.243.129:37243), b8be9b48c8c14cbf9d5cfe28161705dc (127.3.243.130:44043)
I20260619 04:50:19.680647 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.674686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:51545 (local address 127.3.243.130:44043)
0619 04:50:19.674877 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.674883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.674904 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:50:19.675014 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.675022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.675134 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.675233 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:19.675785 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.676781 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.678660 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.679687 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.679875 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.679925 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.680269 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.680393 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 04:50:19.680789 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.674484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:44043 (local address 127.3.243.131:51545)
0619 04:50:19.674723 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:19.674749 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:19.674875 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:19.675348 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:19.675353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:19.675396 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:19.675576 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.675586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.676940 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.676946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.678505 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.678516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.679889 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.679896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.680066 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.680102 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 04:50:19.680248 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 04:50:19.680399 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1}
I20260619 04:50:19.681303 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.674810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:42247 (local address 127.3.243.129:37243)
0619 04:50:19.675091 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.675099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.675570 (+ 471us) server_negotiation.cc:407] Connection header received
0619 04:50:19.675750 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.675757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.675841 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.675909 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:19.676554 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.677490 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.679327 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.680295 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.680433 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.680500 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.680942 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.681109 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 04:50:19.681803 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.674555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:37243 (local address 127.3.243.131:42247)
0619 04:50:19.675470 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:19.675483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:19.675585 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:19.675950 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:19.675954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:19.675978 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:19.676389 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.676400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.677628 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.677632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.679125 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:19.679138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.680448 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.680454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:19.680664 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.680728 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 04:50:19.681488 (+ 760us) client_negotiation.cc:240] Negotiation successful
0619 04:50:19.681604 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":587,"threads_started":1}
I20260619 04:50:19.681830 23113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc43894222f470a8b023e7f61872faf" candidate_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" is_pre_election: true
I20260619 04:50:19.682478 23113 raft_consensus.cc:2454] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a65fbaa5b484a398e354d86d0f1ea2f in term 0.
I20260619 04:50:19.682854 22982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc43894222f470a8b023e7f61872faf" candidate_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" is_pre_election: true
I20260619 04:50:19.683135 23176 leader_election.cc:302] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a65fbaa5b484a398e354d86d0f1ea2f, b8be9b48c8c14cbf9d5cfe28161705dc; no voters:
I20260619 04:50:19.683405 22982 raft_consensus.cc:2454] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a65fbaa5b484a398e354d86d0f1ea2f in term 0.
I20260619 04:50:19.683573 23313 raft_consensus.cc:2781] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:50:19.683713 23313 raft_consensus.cc:482] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:50:19.683820 23313 raft_consensus.cc:3037] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:19.686084 23313 raft_consensus.cc:504] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } }
I20260619 04:50:19.686642 23313 leader_election.cc:288] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [CANDIDATE]: Term 1 election: Requested vote from peers a0d3a6d88a8e4502844faf0dc7547f0b (127.3.243.129:37243), b8be9b48c8c14cbf9d5cfe28161705dc (127.3.243.130:44043)
I20260619 04:50:19.687135 23113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc43894222f470a8b023e7f61872faf" candidate_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc"
I20260619 04:50:19.687135 22982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc43894222f470a8b023e7f61872faf" candidate_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b"
I20260619 04:50:19.687338 22982 raft_consensus.cc:3037] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:19.687348 23113 raft_consensus.cc:3037] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:19.690572 23113 raft_consensus.cc:2454] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a65fbaa5b484a398e354d86d0f1ea2f in term 1.
I20260619 04:50:19.690570 22982 raft_consensus.cc:2454] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a65fbaa5b484a398e354d86d0f1ea2f in term 1.
I20260619 04:50:19.691064 23176 leader_election.cc:302] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a65fbaa5b484a398e354d86d0f1ea2f, b8be9b48c8c14cbf9d5cfe28161705dc; no voters:
I20260619 04:50:19.691329 23313 raft_consensus.cc:2781] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:19.691758 23313 raft_consensus.cc:686] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 1 LEADER]: Becoming Leader. State: Replica: 1a65fbaa5b484a398e354d86d0f1ea2f, State: Running, Role: LEADER
I20260619 04:50:19.692276 23313 consensus_queue.cc:237] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } }
I20260619 04:50:19.696631 22841 catalog_manager.cc:5495] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a65fbaa5b484a398e354d86d0f1ea2f (127.3.243.131). New cstate: current_term: 1 leader_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a65fbaa5b484a398e354d86d0f1ea2f" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 } health_report { overall_health: UNKNOWN } } }
I20260619 04:50:19.718341 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.713238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.3.243.131:36121)
0619 04:50:19.713406 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.713410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.713425 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:50:19.713496 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.713499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.713573 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.713687 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:50:19.714057 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.714758 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.716235 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.716653 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.716744 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.717263 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:50:19.717723 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:50:19.717843 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.718000 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.718062 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 04:50:19.723371 23159 tablet.cc:2369] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:19.726994 22982 raft_consensus.cc:1261] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [term 1 FOLLOWER]: Refusing update from remote peer 1a65fbaa5b484a398e354d86d0f1ea2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:19.726994 23113 raft_consensus.cc:1261] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [term 1 FOLLOWER]: Refusing update from remote peer 1a65fbaa5b484a398e354d86d0f1ea2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:19.728039 23316 consensus_queue.cc:1035] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8be9b48c8c14cbf9d5cfe28161705dc" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:19.728451 23313 consensus_queue.cc:1035] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d3a6d88a8e4502844faf0dc7547f0b" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 37243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:19.740888 23321 mvcc.cc:205] Tried to move back new op lower bound from 7298435562390327296 to 7298435562263572480. Current Snapshot: MvccSnapshot[applied={T|T < 7298435562390327296}]
I20260619 04:50:19.741743 23320 mvcc.cc:205] Tried to move back new op lower bound from 7298435562390327296 to 7298435562263572480. Current Snapshot: MvccSnapshot[applied={T|T < 7298435562390327296}]
I20260619 04:50:19.757212 22840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:19.757575 22840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:19.761898 22840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40cfa469754a49bfb5ecf5a9a7428eaf: Sending DeleteTablet for 3 replicas of tablet 6cc43894222f470a8b023e7f61872faf
I20260619 04:50:19.763141 22962 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc43894222f470a8b023e7f61872faf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:19 UTC) from {username='slave'} at 127.0.0.1:45220
I20260619 04:50:19.763257 23093 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc43894222f470a8b023e7f61872faf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:19 UTC) from {username='slave'} at 127.0.0.1:60398
I20260619 04:50:19.763331 23223 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc43894222f470a8b023e7f61872faf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:19 UTC) from {username='slave'} at 127.0.0.1:59898
I20260619 04:50:19.765201 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 22900
I20260619 04:50:19.776660 23332 tablet_replica.cc:330] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: stopping tablet replica
I20260619 04:50:19.777148 23332 raft_consensus.cc:2229] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:50:19.777946 23332 raft_consensus.cc:2258] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:19.780143 23332 ts_tablet_manager.cc:1905] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:19.787873 23332 ts_tablet_manager.cc:1918] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:19.788157 23332 log.cc:1196] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/6cc43894222f470a8b023e7f61872faf
I20260619 04:50:19.788784 23332 ts_tablet_manager.cc:1939] T 6cc43894222f470a8b023e7f61872faf P 1a65fbaa5b484a398e354d86d0f1ea2f: Deleting consensus metadata
I20260619 04:50:19.790763 22825 catalog_manager.cc:4841] TS 1a65fbaa5b484a398e354d86d0f1ea2f (127.3.243.131:36121): tablet 6cc43894222f470a8b023e7f61872faf (table test-table [id=e67ca6d534df4de18442e86fa8fd8537]) successfully deleted
I20260619 04:50:19.791852 23331 tablet_replica.cc:330] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: stopping tablet replica
I20260619 04:50:19.792344 23331 raft_consensus.cc:2229] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:19.792884 23331 raft_consensus.cc:2258] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:19.794301 23330 tablet_replica.cc:330] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: stopping tablet replica
I20260619 04:50:19.794741 23330 raft_consensus.cc:2229] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:19.795167 23331 ts_tablet_manager.cc:1905] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:19.795250 23330 raft_consensus.cc:2258] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:19.797339 23330 ts_tablet_manager.cc:1905] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:19.800906 23331 ts_tablet_manager.cc:1918] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:19.801076 23331 log.cc:1196] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/6cc43894222f470a8b023e7f61872faf
I20260619 04:50:19.801597 23331 ts_tablet_manager.cc:1939] T 6cc43894222f470a8b023e7f61872faf P a0d3a6d88a8e4502844faf0dc7547f0b: Deleting consensus metadata
I20260619 04:50:19.803103 22826 catalog_manager.cc:4841] TS a0d3a6d88a8e4502844faf0dc7547f0b (127.3.243.129:37243): tablet 6cc43894222f470a8b023e7f61872faf (table test-table [id=e67ca6d534df4de18442e86fa8fd8537]) successfully deleted
I20260619 04:50:19.804185 23330 ts_tablet_manager.cc:1918] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:19.804360 23330 log.cc:1196] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/6cc43894222f470a8b023e7f61872faf
I20260619 04:50:19.804877 23330 ts_tablet_manager.cc:1939] T 6cc43894222f470a8b023e7f61872faf P b8be9b48c8c14cbf9d5cfe28161705dc: Deleting consensus metadata
I20260619 04:50:19.805534 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.767295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45224 (local address 127.3.243.129:37243)
0619 04:50:19.767533 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.767542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.791223 (+ 23681us) server_negotiation.cc:407] Connection header received
0619 04:50:19.791386 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.791395 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.791536 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.791671 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:19.792684 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.794414 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.796228 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.796919 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.797020 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.803893 (+ 6873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:19.803904 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:19.803931 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:19.804679 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:19.804729 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:19.804748 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:19.804869 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.805052 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.805173 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 04:50:19.806293 23022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:19.806559 22825 catalog_manager.cc:4841] TS b8be9b48c8c14cbf9d5cfe28161705dc (127.3.243.130:44043): tablet 6cc43894222f470a8b023e7f61872faf (table test-table [id=e67ca6d534df4de18442e86fa8fd8537]) successfully deleted
I20260619 04:50:19.920073 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 22900
I20260619 04:50:19.937449 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 23031
I20260619 04:50:19.946425 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:19.938293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.3.243.130:44043)
0619 04:50:19.938518 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:19.938525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:19.938547 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:50:19.938621 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:19.938626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:19.938732 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:19.938835 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:19.939196 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.940427 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.942301 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:19.942958 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:19.943080 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:19.944477 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:19.944490 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:19.944511 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:19.945312 (+ 801us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:19.945386 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:19.945410 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:19.945528 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:19.945960 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 04:50:19.946068 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 04:50:19.947116 23153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:20.057471 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 23031
I20260619 04:50:20.075186 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 23161
I20260619 04:50:20.083191 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:20.076130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.3.243.131:36121)
0619 04:50:20.076299 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:20.076303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:20.076322 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:50:20.076565 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:20.076569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:20.076660 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:20.076759 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:20.077276 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:20.078068 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:20.079928 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:20.080432 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:20.080522 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:20.081706 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:20.081714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:20.081729 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:20.082370 (+ 641us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:20.082416 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:20.082435 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:20.082515 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:20.082702 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 04:50:20.082799 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 04:50:20.083976 23283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:20.208922 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 23161
I20260619 04:50:20.228154 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 22810
I20260619 04:50:20.229192 22871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:20.336794 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 22810
2026-06-19T04:50:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2886 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:20 dist-test-slave-2qp8 krb5kdc[23340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:20 dist-test-slave-2qp8 krb5kdc[23340](info): set up 2 sockets
Jun 19 04:50:20 dist-test-slave-2qp8 krb5kdc[23340](info): commencing operation
krb5kdc: starting...
W20260619 04:50:21.428553 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:21 dist-test-slave-2qp8 krb5kdc[23340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:50:21.675132 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:38047
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:38047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:21.998967 23353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:21.999241 23353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:21.999408 23353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:22.006551 23353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:22.006659 23353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:22.006770 23353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:22.006872 23353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:22.006946 23353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:22.016711 23353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:38047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:38047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:22.018039 23353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:22.019349 23353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:22.031458 23359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:22.031522 23358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:22.032368 23361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:22.032392 23353 server_base.cc:1034] running on GCE node
I20260619 04:50:22.033215 23353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:22.034641 23353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:22.035866 23353 hybrid_clock.cc:648] HybridClock initialized: now 1781844622035812 us; error 43 us; skew 500 ppm
I20260619 04:50:22.036281 23353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:22 dist-test-slave-2qp8 krb5kdc[23340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:22.042655 23353 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:50:22.044306 23353 webserver.cc:460] Webserver started at http://127.3.243.190:46753/ using document root <none> and password file <none>
I20260619 04:50:22.044997 23353 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:22.045127 23353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:22.045451 23353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:22.048472 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "4509929e76574bb4b72e3f55bda23bdf"
format_stamp: "Formatted at 2026-06-19 04:50:22 on dist-test-slave-2qp8"
server_key: "0515e03a73d4569d10386e44a6a1aeb1"
server_key_iv: "855be880b6f98c4e2457bd4b851cbeb7"
server_key_version: "encryptionkey@0"
I20260619 04:50:22.049216 23353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "4509929e76574bb4b72e3f55bda23bdf"
format_stamp: "Formatted at 2026-06-19 04:50:22 on dist-test-slave-2qp8"
server_key: "0515e03a73d4569d10386e44a6a1aeb1"
server_key_iv: "855be880b6f98c4e2457bd4b851cbeb7"
server_key_version: "encryptionkey@0"
I20260619 04:50:22.054216 23353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:50:22.057618 23368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:22.058887 23353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:50:22.059077 23353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "4509929e76574bb4b72e3f55bda23bdf"
format_stamp: "Formatted at 2026-06-19 04:50:22 on dist-test-slave-2qp8"
server_key: "0515e03a73d4569d10386e44a6a1aeb1"
server_key_iv: "855be880b6f98c4e2457bd4b851cbeb7"
server_key_version: "encryptionkey@0"
I20260619 04:50:22.059286 23353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:22.078406 23353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:22.082742 23353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:22.083019 23353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:22.101955 23353 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:38047
I20260619 04:50:22.101974 23419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:38047 every 8 connection(s)
I20260619 04:50:22.103377 23353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:22.107115 23420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:22.108201 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 23353
I20260619 04:50:22.108418 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:22.108861 4046 external_mini_cluster.cc:1428] Setting key 2f3fca1059fe7cb73a12446e8c8b849b
I20260619 04:50:22.121340 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf: Bootstrap starting.
Jun 19 04:50:22 dist-test-slave-2qp8 krb5kdc[23340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:22.125385 23420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:22.126760 23420 log.cc:824] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:22.130189 23420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf: No bootstrap required, opened a new log
I20260619 04:50:22.134250 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:22.110787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.3.243.190:38047)
0619 04:50:22.111642 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:22.111653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:22.111710 (+ 57us) server_negotiation.cc:407] Connection header received
0619 04:50:22.113413 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:22.113438 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:22.113836 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:22.114165 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:22.116280 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:22.117564 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:22.119397 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:22.120054 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:22.120348 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:22.125424 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:22.125436 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:22.125457 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:22.125509 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:22.129467 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:22.130244 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:22.130252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:22.130260 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:22.130474 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:22.130877 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:22.130883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:22.130886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:22.131614 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:22.131939 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:22.132345 (+ 406us) server_negotiation.cc:299] Negotiation successful
0619 04:50:22.132796 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":387,"threads_started":1}
I20260619 04:50:22.139297 23420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4509929e76574bb4b72e3f55bda23bdf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38047 } }
I20260619 04:50:22.139657 23420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:22.139808 23420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4509929e76574bb4b72e3f55bda23bdf, State: Initialized, Role: FOLLOWER
I20260619 04:50:22.140421 23420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4509929e76574bb4b72e3f55bda23bdf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38047 } }
I20260619 04:50:22.140656 23420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:22.140803 23420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:22.140966 23420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:22.143625 23420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4509929e76574bb4b72e3f55bda23bdf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38047 } }
I20260619 04:50:22.144255 23420 leader_election.cc:302] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4509929e76574bb4b72e3f55bda23bdf; no voters:
I20260619 04:50:22.144832 23420 leader_election.cc:288] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:22.144992 23425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:22.145697 23425 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [term 1 LEADER]: Becoming Leader. State: Replica: 4509929e76574bb4b72e3f55bda23bdf, State: Running, Role: LEADER
I20260619 04:50:22.146198 23425 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4509929e76574bb4b72e3f55bda23bdf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38047 } }
I20260619 04:50:22.146926 23420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:22.149222 23427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4509929e76574bb4b72e3f55bda23bdf. Latest consensus state: current_term: 1 leader_uuid: "4509929e76574bb4b72e3f55bda23bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4509929e76574bb4b72e3f55bda23bdf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38047 } } }
I20260619 04:50:22.149634 23427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:22.152292 23426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4509929e76574bb4b72e3f55bda23bdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4509929e76574bb4b72e3f55bda23bdf" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38047 } } }
I20260619 04:50:22.152554 23426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:22.153081 23433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:22.158069 23433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:22.167310 23433 catalog_manager.cc:1347] Generated new cluster ID: 01a798acc5f1475b884b8a096253b87f
I20260619 04:50:22.167436 23433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:22.183069 23433 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:22.185184 23433 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:22.197222 23433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf: Generated new TSK 0
I20260619 04:50:22.198436 23433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
Entry for principal kudu/127.3.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab.
I20260619 04:50:22.269713 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:38047
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:22.597792 23446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:22.598136 23446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:22.598316 23446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:22.605660 23446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:22.605829 23446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:22.606020 23446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:50:22.615516 23446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.129.keytab
--principal=kudu/127.3.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:38047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:22.616880 23446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:22.618248 23446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:22.631374 23451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:22.631326 23452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:22.631404 23454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:22.632158 23446 server_base.cc:1034] running on GCE node
I20260619 04:50:22.632843 23446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:22.633787 23446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:22.635068 23446 hybrid_clock.cc:648] HybridClock initialized: now 1781844622634998 us; error 63 us; skew 500 ppm
I20260619 04:50:22.635435 23446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:22 dist-test-slave-2qp8 krb5kdc[23340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:22.638586 23446 init.cc:377] Logged in from keytab as kudu/127.3.243.129@KRBTEST.COM (short username kudu)
I20260619 04:50:22.640012 23446 webserver.cc:460] Webserver started at http://127.3.243.129:38409/ using document root <none> and password file <none>
I20260619 04:50:22.640703 23446 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:22.640851 23446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:22.641176 23446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:22.644030 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "45a21a5f5f354af18836ebbe61a5a1ef"
format_stamp: "Formatted at 2026-06-19 04:50:22 on dist-test-slave-2qp8"
server_key: "524352ea1110869580dfbc32974e5559"
server_key_iv: "762678ba1795b8ee37c655c363bc7b42"
server_key_version: "encryptionkey@0"
I20260619 04:50:22.644871 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "45a21a5f5f354af18836ebbe61a5a1ef"
format_stamp: "Formatted at 2026-06-19 04:50:22 on dist-test-slave-2qp8"
server_key: "524352ea1110869580dfbc32974e5559"
server_key_iv: "762678ba1795b8ee37c655c363bc7b42"
server_key_version: "encryptionkey@0"
I20260619 04:50:22.650051 23446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 04:50:22.653383 23461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:22.654597 23446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:50:22.654774 23446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "45a21a5f5f354af18836ebbe61a5a1ef"
format_stamp: "Formatted at 2026-06-19 04:50:22 on dist-test-slave-2qp8"
server_key: "524352ea1110869580dfbc32974e5559"
server_key_iv: "762678ba1795b8ee37c655c363bc7b42"
server_key_version: "encryptionkey@0"
I20260619 04:50:22.654975 23446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:22.670691 23446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:22.674463 23446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:22.674750 23446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:22.675711 23446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:22.677106 23446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:22.677206 23446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:22.677358 23446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:22.677433 23446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:22.712083 23446 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:39947
I20260619 04:50:22.712178 23573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:39947 every 8 connection(s)
I20260619 04:50:22.713550 23446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:50:22.716513 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 23446
I20260619 04:50:22.716691 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:50:22.717235 4046 external_mini_cluster.cc:1428] Setting key 786978c03b3aacbfaaf59618bd647f73
Jun 19 04:50:22 dist-test-slave-2qp8 krb5kdc[23340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844622, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.129@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:22.734669 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:22.716080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:58381 (local address 127.3.243.190:38047)
0619 04:50:22.716604 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:22.716610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:22.717795 (+ 1185us) server_negotiation.cc:407] Connection header received
0619 04:50:22.719820 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:22.719826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:22.719909 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:22.720030 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:22.721516 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:22.722650 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:22.724265 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:22.725021 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:22.725169 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:22.728784 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:22.728789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:22.728793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:22.728823 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:22.731547 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:22.732372 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:22.732375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:22.732378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:22.732496 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:22.732961 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:22.732964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:22.732966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:22.733233 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:22.733391 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:22.734080 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 04:50:22.734307 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":235,"threads_started":1}
I20260619 04:50:22.735610 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:22.716722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:38047 (local address 127.3.243.129:58381)
0619 04:50:22.717603 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:22.717660 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:22.719544 (+ 1884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:22.720274 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:22.720281 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:22.720763 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:22.721351 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:22.721367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:22.722810 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:22.722813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:22.724076 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:22.724086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:22.725234 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:22.725238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:22.725431 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:22.726110 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:50:22.726131 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:22.728564 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:22.731693 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:22.731701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:22.731717 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:22.732210 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:22.732609 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:22.732614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:22.732618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:22.732811 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:22.733362 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:22.733369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:22.733707 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 04:50:22.733977 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 04:50:22.734509 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":304,"threads_started":1}
I20260619 04:50:22.737221 23574 heartbeater.cc:347] Connected to a master server at 127.3.243.190:38047
I20260619 04:50:22.737543 23574 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:22.738287 23574 heartbeater.cc:511] Master 127.3.243.190:38047 requested a full tablet report, sending...
I20260619 04:50:22.740456 23385 ts_manager.cc:194] Registered new tserver with Master: 45a21a5f5f354af18836ebbe61a5a1ef (127.3.243.129:39947)
WARNING: no policy specified for kudu/127.3.243.130@KRBTEST.COM; defaulting to no policy
I20260619 04:50:22.741936 23385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.129@KRBTEST.COM'} at 127.3.243.129:58381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
Entry for principal kudu/127.3.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab.
I20260619 04:50:22.773609 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:38047
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:23.095033 23580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:23.095294 23580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:23.095496 23580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:23.102821 23580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:23.102926 23580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:23.103089 23580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:50:23.112896 23580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.130.keytab
--principal=kudu/127.3.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:38047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:23.114069 23580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:23.115370 23580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:23.128669 23585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:23.128674 23586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:23.128697 23588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:23.129765 23580 server_base.cc:1034] running on GCE node
I20260619 04:50:23.130385 23580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:23.131333 23580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:23.132560 23580 hybrid_clock.cc:648] HybridClock initialized: now 1781844623132522 us; error 53 us; skew 500 ppm
I20260619 04:50:23.132879 23580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:23 dist-test-slave-2qp8 krb5kdc[23340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:23.136158 23580 init.cc:377] Logged in from keytab as kudu/127.3.243.130@KRBTEST.COM (short username kudu)
I20260619 04:50:23.137537 23580 webserver.cc:460] Webserver started at http://127.3.243.130:36849/ using document root <none> and password file <none>
I20260619 04:50:23.138209 23580 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:23.138360 23580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:23.138659 23580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:23.141546 23580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "eafd6e81c59c48d990666b02344b9b48"
format_stamp: "Formatted at 2026-06-19 04:50:23 on dist-test-slave-2qp8"
server_key: "3c4252c9bfd5b473d7679361b2fa1f04"
server_key_iv: "ff7850bd44bee06d0c8a1be9602d77bd"
server_key_version: "encryptionkey@0"
I20260619 04:50:23.142349 23580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "eafd6e81c59c48d990666b02344b9b48"
format_stamp: "Formatted at 2026-06-19 04:50:23 on dist-test-slave-2qp8"
server_key: "3c4252c9bfd5b473d7679361b2fa1f04"
server_key_iv: "ff7850bd44bee06d0c8a1be9602d77bd"
server_key_version: "encryptionkey@0"
I20260619 04:50:23.147338 23580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 04:50:23.150799 23595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:23.152244 23580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:23.152439 23580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "eafd6e81c59c48d990666b02344b9b48"
format_stamp: "Formatted at 2026-06-19 04:50:23 on dist-test-slave-2qp8"
server_key: "3c4252c9bfd5b473d7679361b2fa1f04"
server_key_iv: "ff7850bd44bee06d0c8a1be9602d77bd"
server_key_version: "encryptionkey@0"
I20260619 04:50:23.152648 23580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:23.165961 23580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:23.170219 23580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:23.170516 23580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:23.171420 23580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:23.172806 23580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:23.172904 23580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:23.173041 23580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:23.173113 23580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:23.204923 23580 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:33695
I20260619 04:50:23.204977 23707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:33695 every 8 connection(s)
I20260619 04:50:23.206187 23580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:50:23.206678 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 23580
I20260619 04:50:23.206893 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:50:23.207442 4046 external_mini_cluster.cc:1428] Setting key 166878e395ff9e59fd4db94b98d0352e
Jun 19 04:50:23 dist-test-slave-2qp8 krb5kdc[23340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.130@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:23.227016 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.208935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:56093 (local address 127.3.243.190:38047)
0619 04:50:23.209105 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:23.209110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:23.210640 (+ 1530us) server_negotiation.cc:407] Connection header received
0619 04:50:23.212556 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:23.212565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:23.212655 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:23.212764 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:23.214203 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.215373 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.216975 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.217616 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.217752 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.221485 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:23.221490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:23.221493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:23.221524 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:23.223990 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:23.224717 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:23.224722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:23.224725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:23.224856 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:23.225295 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:23.225298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:23.225300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:23.225576 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:23.225726 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:23.226458 (+ 732us) server_negotiation.cc:299] Negotiation successful
0619 04:50:23.226679 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:50:23.228083 23710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.209613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:38047 (local address 127.3.243.130:56093)
0619 04:50:23.210462 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:23.210509 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:23.212300 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:23.212963 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:23.212971 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:23.213486 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:23.214031 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.214048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.215513 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.215517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.216834 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.216846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.217837 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.217844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.218071 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.218697 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:50:23.218730 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:23.221267 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:23.224153 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:23.224163 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:23.224178 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:23.224584 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:23.224964 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:23.224970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:23.224973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:23.225166 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:23.225701 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:23.225708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:23.226074 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 04:50:23.226365 (+ 291us) client_negotiation.cc:240] Negotiation successful
0619 04:50:23.226875 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":341,"threads_started":1}
I20260619 04:50:23.229573 23708 heartbeater.cc:347] Connected to a master server at 127.3.243.190:38047
I20260619 04:50:23.229916 23708 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:23.230875 23708 heartbeater.cc:511] Master 127.3.243.190:38047 requested a full tablet report, sending...
I20260619 04:50:23.232553 23385 ts_manager.cc:194] Registered new tserver with Master: eafd6e81c59c48d990666b02344b9b48 (127.3.243.130:33695)
WARNING: no policy specified for kudu/127.3.243.131@KRBTEST.COM; defaulting to no policy
I20260619 04:50:23.233551 23385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.130@KRBTEST.COM'} at 127.3.243.130:56093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
Entry for principal kudu/127.3.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab.
I20260619 04:50:23.264060 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:38047
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:38543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:23.583189 23713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:23.583458 23713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:23.583551 23713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:23.590736 23713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:23.590839 23713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:23.590951 23713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:50:23.600472 23713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:38543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.131.keytab
--principal=kudu/127.3.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:38047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:23.601756 23713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:23.603098 23713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:23.615387 23719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:23.615396 23721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:23.615644 23718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:23.616508 23713 server_base.cc:1034] running on GCE node
I20260619 04:50:23.617027 23713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:23.617859 23713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:23.619056 23713 hybrid_clock.cc:648] HybridClock initialized: now 1781844623619010 us; error 40 us; skew 500 ppm
I20260619 04:50:23.619330 23713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:23 dist-test-slave-2qp8 krb5kdc[23340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:23.622431 23713 init.cc:377] Logged in from keytab as kudu/127.3.243.131@KRBTEST.COM (short username kudu)
I20260619 04:50:23.623917 23713 webserver.cc:460] Webserver started at http://127.3.243.131:36759/ using document root <none> and password file <none>
I20260619 04:50:23.624646 23713 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:23.624771 23713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:23.625159 23713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:23.628221 23713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "c8a9bed721d24057b471dd9629a03818"
format_stamp: "Formatted at 2026-06-19 04:50:23 on dist-test-slave-2qp8"
server_key: "47d24777c05ceb5128aa900272034760"
server_key_iv: "531c3d203fc203ade222a5bd912c409a"
server_key_version: "encryptionkey@0"
I20260619 04:50:23.629004 23713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "c8a9bed721d24057b471dd9629a03818"
format_stamp: "Formatted at 2026-06-19 04:50:23 on dist-test-slave-2qp8"
server_key: "47d24777c05ceb5128aa900272034760"
server_key_iv: "531c3d203fc203ade222a5bd912c409a"
server_key_version: "encryptionkey@0"
I20260619 04:50:23.633885 23713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:50:23.637367 23728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:23.638624 23713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:50:23.638785 23713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "c8a9bed721d24057b471dd9629a03818"
format_stamp: "Formatted at 2026-06-19 04:50:23 on dist-test-slave-2qp8"
server_key: "47d24777c05ceb5128aa900272034760"
server_key_iv: "531c3d203fc203ade222a5bd912c409a"
server_key_version: "encryptionkey@0"
I20260619 04:50:23.638985 23713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:23.660315 23713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:23.664654 23713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:23.664933 23713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:23.665851 23713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:23.667066 23713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:23.667171 23713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:23.667318 23713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:23.667394 23713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:23.700026 23713 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:34785
I20260619 04:50:23.700080 23840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:34785 every 8 connection(s)
I20260619 04:50:23.701437 23713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:50:23.706605 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 23713
I20260619 04:50:23.706804 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:50:23.707249 4046 external_mini_cluster.cc:1428] Setting key 6df86d5dea76c17b0280ba2858296d4a
Jun 19 04:50:23 dist-test-slave-2qp8 krb5kdc[23340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844623, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.131@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:23.721483 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.704169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:49789 (local address 127.3.243.190:38047)
0619 04:50:23.704371 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:23.704376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:23.705505 (+ 1129us) server_negotiation.cc:407] Connection header received
0619 04:50:23.707372 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:23.707380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:23.707474 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:23.707585 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:23.709185 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.710119 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.711650 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.712290 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.712433 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.716084 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:23.716091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:23.716095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:23.716166 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:23.718663 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:23.719270 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:23.719274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:23.719276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:23.719393 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:23.719733 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:23.719736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:23.719738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:23.719987 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:23.720205 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:23.720926 (+ 721us) server_negotiation.cc:299] Negotiation successful
0619 04:50:23.721128 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:50:23.722409 23843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.704647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:38047 (local address 127.3.243.131:49789)
0619 04:50:23.705332 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:23.705376 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:23.707081 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:23.707865 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:23.707877 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:23.708470 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:23.709013 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.709034 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.710266 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.710272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.711470 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.711508 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.712481 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.712486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.712668 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.713341 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:50:23.713362 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:23.715882 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:23.718781 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:23.718787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:23.718797 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:23.719176 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:23.719480 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:50:23.719483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:50:23.719486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:50:23.719646 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:50:23.720174 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:23.720181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:23.720478 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 04:50:23.720815 (+ 337us) client_negotiation.cc:240] Negotiation successful
0619 04:50:23.721300 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
I20260619 04:50:23.723630 23841 heartbeater.cc:347] Connected to a master server at 127.3.243.190:38047
I20260619 04:50:23.723930 23841 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:23.724699 23841 heartbeater.cc:511] Master 127.3.243.190:38047 requested a full tablet report, sending...
I20260619 04:50:23.726032 23385 ts_manager.cc:194] Registered new tserver with Master: c8a9bed721d24057b471dd9629a03818 (127.3.243.131:34785)
I20260619 04:50:23.726816 23385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.243.131@KRBTEST.COM'} at 127.3.243.131:49789
I20260619 04:50:23.734326 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:50:23.745078 23574 heartbeater.cc:503] Master 127.3.243.190:38047 was elected leader, sending a full tablet report...
I20260619 04:50:23.751902 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.738364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.3.243.190:38047)
0619 04:50:23.738523 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:23.738527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:23.739005 (+ 478us) server_negotiation.cc:407] Connection header received
0619 04:50:23.739141 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:23.739145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:23.739211 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:23.739288 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:23.740676 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.741362 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.742754 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.743196 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.743288 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.746158 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:23.746165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:23.746170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:23.746238 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:23.749147 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:23.749827 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:23.749835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:23.749839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:23.749981 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:23.750368 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:23.750376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:23.750380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:23.750787 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:23.751047 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:23.751360 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 04:50:23.751577 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 04:50:23.755462 23385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:50:23.762310 23385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:50:23.785710 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.775666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38854 (local address 127.3.243.129:39947)
0619 04:50:23.778140 (+ 2474us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:23.778149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:23.778172 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:50:23.778259 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:23.778266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:23.778475 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:23.778643 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:23.779387 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.781258 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.783397 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.784555 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.784708 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.784961 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:23.785202 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 04:50:23.785359 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2323,"thread_start_us":2184,"threads_started":1}
I20260619 04:50:23.785878 23854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.775037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:34785 (local address 127.0.0.1:34740)
0619 04:50:23.776388 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:23.776443 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:23.776612 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:23.778411 (+ 1799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:23.778416 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:23.778465 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:23.778916 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.778928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.780731 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.780740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.783036 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.783054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.784916 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.784922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.785116 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.785162 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 04:50:23.785227 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 04:50:23.785347 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":6,"spinlock_wait_cycles":2432,"thread_start_us":1992,"threads_started":1}
I20260619 04:50:23.786324 23857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.775426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:39947 (local address 127.0.0.1:38854)
0619 04:50:23.777015 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:23.777041 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:23.777159 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:23.778660 (+ 1501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:23.778664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:23.778693 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:23.779152 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.779168 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.781448 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.781456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.783204 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.783218 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.784736 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.784742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.784931 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.785010 (+ 79us) client_negotiation.cc:769] Sending connection context
0619 04:50:23.785163 (+ 153us) client_negotiation.cc:240] Negotiation successful
0619 04:50:23.785347 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"mutex_wait_us":3,"thread_start_us":524,"threads_started":1}
I20260619 04:50:23.786473 23853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.776361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:33695 (local address 127.0.0.1:57406)
0619 04:50:23.777206 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:23.777232 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:23.777368 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:23.778407 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:23.778414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:23.778453 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:23.779416 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.779430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.781343 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.781351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.783673 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.783687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.785575 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.785581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.785780 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.785811 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 04:50:23.785913 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 04:50:23.786045 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"mutex_wait_us":73,"spinlock_wait_cycles":1792,"thread_start_us":392,"threads_started":1}
I20260619 04:50:23.786790 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.776512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57406 (local address 127.3.243.130:33695)
0619 04:50:23.777872 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:23.777880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:23.777907 (+ 27us) server_negotiation.cc:407] Connection header received
0619 04:50:23.778008 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:23.778015 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:23.778206 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:23.778411 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:23.779593 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.781195 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.783847 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.785340 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.785469 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.785668 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:23.786282 (+ 614us) server_negotiation.cc:299] Negotiation successful
0619 04:50:23.786456 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1200,"thread_start_us":1074,"threads_started":1}
I20260619 04:50:23.787350 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.775704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.3.243.131:34785)
0619 04:50:23.777769 (+ 2065us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:23.777781 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:23.777817 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:50:23.777919 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:23.777927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:23.778184 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:23.778554 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:23.779129 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.780607 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.783243 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.784682 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.786573 (+ 1891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.786816 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:23.786879 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 04:50:23.787022 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1867,"thread_start_us":1709,"threads_started":1}
I20260619 04:50:23.789909 23509 tablet_service.cc:1467] Processing CreateTablet for tablet 73d3db734b8d4a7db6076d2e4ab2ad4a (DEFAULT_TABLE table=test-table [id=e28f5d996909406586c402e76cb4b299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:23.790445 23643 tablet_service.cc:1467] Processing CreateTablet for tablet 73d3db734b8d4a7db6076d2e4ab2ad4a (DEFAULT_TABLE table=test-table [id=e28f5d996909406586c402e76cb4b299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:23.791191 23509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d3db734b8d4a7db6076d2e4ab2ad4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:23.790756 23776 tablet_service.cc:1467] Processing CreateTablet for tablet 73d3db734b8d4a7db6076d2e4ab2ad4a (DEFAULT_TABLE table=test-table [id=e28f5d996909406586c402e76cb4b299]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:23.791836 23643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d3db734b8d4a7db6076d2e4ab2ad4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:23.792025 23776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d3db734b8d4a7db6076d2e4ab2ad4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:23.801987 23859 tablet_bootstrap.cc:492] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: Bootstrap starting.
I20260619 04:50:23.805305 23859 tablet_bootstrap.cc:654] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:23.806361 23859 log.cc:824] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:23.807714 23861 tablet_bootstrap.cc:492] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: Bootstrap starting.
I20260619 04:50:23.809604 23859 tablet_bootstrap.cc:492] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: No bootstrap required, opened a new log
I20260619 04:50:23.809598 23860 tablet_bootstrap.cc:492] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: Bootstrap starting.
I20260619 04:50:23.809913 23859 ts_tablet_manager.cc:1397] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 04:50:23.812393 23860 tablet_bootstrap.cc:654] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:23.812649 23861 tablet_bootstrap.cc:654] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:23.813493 23860 log.cc:824] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:23.813776 23861 log.cc:824] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:23.816268 23860 tablet_bootstrap.cc:492] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: No bootstrap required, opened a new log
I20260619 04:50:23.816509 23860 ts_tablet_manager.cc:1397] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 04:50:23.816812 23861 tablet_bootstrap.cc:492] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: No bootstrap required, opened a new log
I20260619 04:50:23.817056 23861 ts_tablet_manager.cc:1397] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260619 04:50:23.820938 23859 raft_consensus.cc:348] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } }
I20260619 04:50:23.821352 23859 raft_consensus.cc:374] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:23.821485 23859 raft_consensus.cc:729] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a21a5f5f354af18836ebbe61a5a1ef, State: Initialized, Role: FOLLOWER
I20260619 04:50:23.822115 23859 consensus_queue.cc:260] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } }
I20260619 04:50:23.823803 23859 ts_tablet_manager.cc:1428] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260619 04:50:23.824083 23860 raft_consensus.cc:348] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } }
I20260619 04:50:23.824546 23860 raft_consensus.cc:374] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:23.824683 23860 raft_consensus.cc:729] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a9bed721d24057b471dd9629a03818, State: Initialized, Role: FOLLOWER
I20260619 04:50:23.825369 23860 consensus_queue.cc:260] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } }
I20260619 04:50:23.826440 23841 heartbeater.cc:503] Master 127.3.243.190:38047 was elected leader, sending a full tablet report...
I20260619 04:50:23.827102 23861 raft_consensus.cc:348] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } }
I20260619 04:50:23.827913 23861 raft_consensus.cc:374] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:23.828047 23861 raft_consensus.cc:729] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eafd6e81c59c48d990666b02344b9b48, State: Initialized, Role: FOLLOWER
I20260619 04:50:23.828087 23860 ts_tablet_manager.cc:1428] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260619 04:50:23.828665 23861 consensus_queue.cc:260] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } }
I20260619 04:50:23.829784 23708 heartbeater.cc:503] Master 127.3.243.190:38047 was elected leader, sending a full tablet report...
I20260619 04:50:23.830359 23861 ts_tablet_manager.cc:1428] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
W20260619 04:50:23.953235 23842 tablet.cc:2369] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:50:23.958343 23709 tablet.cc:2369] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 04:50:23.965826 23575 tablet.cc:2369] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:23.983623 23867 raft_consensus.cc:482] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:50:23.983942 23867 raft_consensus.cc:504] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } }
I20260619 04:50:23.985849 23867 leader_election.cc:288] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a9bed721d24057b471dd9629a03818 (127.3.243.131:34785), 45a21a5f5f354af18836ebbe61a5a1ef (127.3.243.129:39947)
I20260619 04:50:23.992296 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.986542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:33875 (local address 127.3.243.129:39947)
0619 04:50:23.986754 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:23.986762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:23.987164 (+ 402us) server_negotiation.cc:407] Connection header received
0619 04:50:23.987396 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:23.987400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:23.987457 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:23.987549 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:23.988140 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.988855 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.990606 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.991370 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.991551 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.991628 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:23.991907 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 04:50:23.992023 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:50:23.992455 23868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.986150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:39947 (local address 127.3.243.130:33875)
0619 04:50:23.987043 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:23.987065 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:23.987246 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:23.987573 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:23.987578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:23.987602 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:23.987971 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.987980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.988987 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.988995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.990454 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.990468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.991576 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.991582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.991744 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.991767 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 04:50:23.991871 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 04:50:23.992003 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":1,"thread_start_us":548,"threads_started":1}
I20260619 04:50:23.992830 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.986232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:48945 (local address 127.3.243.131:34785)
0619 04:50:23.986461 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:23.986470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:23.987278 (+ 808us) server_negotiation.cc:407] Connection header received
0619 04:50:23.987502 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:23.987505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:23.987565 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:23.987668 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:23.988377 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.989180 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.990992 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.991816 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.991994 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.992073 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:23.992431 (+ 358us) server_negotiation.cc:299] Negotiation successful
0619 04:50:23.992561 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:50:23.992947 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:23.986229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:34785 (local address 127.3.243.130:48945)
0619 04:50:23.987150 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:23.987166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:23.987347 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:23.987683 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:23.987688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:23.987727 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:23.988216 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.988226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.989328 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.989334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.990821 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:23.990832 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:23.992038 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:23.992045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:23.992264 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:23.992289 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 04:50:23.992397 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 04:50:23.992531 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":3,"thread_start_us":575,"threads_started":1}
I20260619 04:50:23.993567 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d3db734b8d4a7db6076d2e4ab2ad4a" candidate_uuid: "eafd6e81c59c48d990666b02344b9b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" is_pre_election: true
I20260619 04:50:23.993937 23796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d3db734b8d4a7db6076d2e4ab2ad4a" candidate_uuid: "eafd6e81c59c48d990666b02344b9b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a9bed721d24057b471dd9629a03818" is_pre_election: true
I20260619 04:50:23.994087 23529 raft_consensus.cc:2454] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafd6e81c59c48d990666b02344b9b48 in term 0.
I20260619 04:50:23.994415 23796 raft_consensus.cc:2454] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eafd6e81c59c48d990666b02344b9b48 in term 0.
I20260619 04:50:23.994874 23597 leader_election.cc:302] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45a21a5f5f354af18836ebbe61a5a1ef, eafd6e81c59c48d990666b02344b9b48; no voters:
I20260619 04:50:23.995302 23867 raft_consensus.cc:2781] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:50:23.995460 23867 raft_consensus.cc:482] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:50:23.995582 23867 raft_consensus.cc:3037] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:23.997956 23867 raft_consensus.cc:504] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } }
I20260619 04:50:23.998597 23867 leader_election.cc:288] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [CANDIDATE]: Term 1 election: Requested vote from peers c8a9bed721d24057b471dd9629a03818 (127.3.243.131:34785), 45a21a5f5f354af18836ebbe61a5a1ef (127.3.243.129:39947)
I20260619 04:50:23.998904 23796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d3db734b8d4a7db6076d2e4ab2ad4a" candidate_uuid: "eafd6e81c59c48d990666b02344b9b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a9bed721d24057b471dd9629a03818"
I20260619 04:50:23.999099 23796 raft_consensus.cc:3037] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:23.999109 23529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d3db734b8d4a7db6076d2e4ab2ad4a" candidate_uuid: "eafd6e81c59c48d990666b02344b9b48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45a21a5f5f354af18836ebbe61a5a1ef"
I20260619 04:50:23.999356 23529 raft_consensus.cc:3037] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:24.001338 23796 raft_consensus.cc:2454] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafd6e81c59c48d990666b02344b9b48 in term 1.
I20260619 04:50:24.001741 23598 leader_election.cc:302] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8a9bed721d24057b471dd9629a03818, eafd6e81c59c48d990666b02344b9b48; no voters:
I20260619 04:50:24.002010 23867 raft_consensus.cc:2781] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:24.002014 23529 raft_consensus.cc:2454] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eafd6e81c59c48d990666b02344b9b48 in term 1.
I20260619 04:50:24.002588 23867 raft_consensus.cc:686] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 1 LEADER]: Becoming Leader. State: Replica: eafd6e81c59c48d990666b02344b9b48, State: Running, Role: LEADER
I20260619 04:50:24.003068 23867 consensus_queue.cc:237] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } }
I20260619 04:50:24.007371 23385 catalog_manager.cc:5495] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 reported cstate change: term changed from 0 to 1, leader changed from <none> to eafd6e81c59c48d990666b02344b9b48 (127.3.243.130). New cstate: current_term: 1 leader_uuid: "eafd6e81c59c48d990666b02344b9b48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eafd6e81c59c48d990666b02344b9b48" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 33695 } health_report { overall_health: HEALTHY } } }
I20260619 04:50:24.079802 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:24.074421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57414 (local address 127.3.243.130:33695)
0619 04:50:24.074630 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:24.074637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:24.074730 (+ 93us) server_negotiation.cc:407] Connection header received
0619 04:50:24.074981 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:24.074985 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:24.075063 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:24.075215 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:50:24.075567 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:24.076338 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:24.077707 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:24.078119 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:24.078255 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:24.078732 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:50:24.079231 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:50:24.079412 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:24.079460 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 04:50:24.079539 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:50:24.088073 23529 raft_consensus.cc:1261] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [term 1 FOLLOWER]: Refusing update from remote peer eafd6e81c59c48d990666b02344b9b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:24.088073 23796 raft_consensus.cc:1261] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [term 1 FOLLOWER]: Refusing update from remote peer eafd6e81c59c48d990666b02344b9b48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:24.089162 23870 consensus_queue.cc:1035] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a9bed721d24057b471dd9629a03818" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:24.089505 23867 consensus_queue.cc:1035] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45a21a5f5f354af18836ebbe61a5a1ef" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:24.098865 23873 mvcc.cc:205] Tried to move back new op lower bound from 7298435580253880320 to 7298435579920453632. Current Snapshot: MvccSnapshot[applied={T|T < 7298435580253880320}]
I20260619 04:50:24.101243 23875 mvcc.cc:205] Tried to move back new op lower bound from 7298435580253880320 to 7298435579920453632. Current Snapshot: MvccSnapshot[applied={T|T < 7298435580253880320}]
I20260619 04:50:24.130441 23385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:24.130764 23385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:24.134415 23385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4509929e76574bb4b72e3f55bda23bdf: Sending DeleteTablet for 3 replicas of tablet 73d3db734b8d4a7db6076d2e4ab2ad4a
I20260619 04:50:24.135552 23776 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d3db734b8d4a7db6076d2e4ab2ad4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:24 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:34740
I20260619 04:50:24.135672 23509 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d3db734b8d4a7db6076d2e4ab2ad4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:24 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:38854
I20260619 04:50:24.135896 23643 tablet_service.cc:1514] Processing DeleteTablet for tablet 73d3db734b8d4a7db6076d2e4ab2ad4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:24 UTC) from {username='kudu', principal='kudu/127.3.243.190@KRBTEST.COM'} at 127.0.0.1:57406
I20260619 04:50:24.137873 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 23446
I20260619 04:50:24.145331 23883 tablet_replica.cc:330] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: stopping tablet replica
I20260619 04:50:24.145907 23883 raft_consensus.cc:2229] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:24.146523 23883 raft_consensus.cc:2258] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:24.147436 23882 tablet_replica.cc:330] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: stopping tablet replica
I20260619 04:50:24.148134 23882 raft_consensus.cc:2229] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:24.148345 23884 tablet_replica.cc:330] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: stopping tablet replica
I20260619 04:50:24.148636 23882 raft_consensus.cc:2258] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:24.148758 23884 raft_consensus.cc:2229] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:50:24.149331 23883 ts_tablet_manager.cc:1905] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:24.149689 23884 raft_consensus.cc:2258] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:24.150898 23882 ts_tablet_manager.cc:1905] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:24.151611 23884 ts_tablet_manager.cc:1905] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:24.156546 23883 ts_tablet_manager.cc:1918] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:24.156749 23883 log.cc:1196] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/73d3db734b8d4a7db6076d2e4ab2ad4a
I20260619 04:50:24.157296 23883 ts_tablet_manager.cc:1939] T 73d3db734b8d4a7db6076d2e4ab2ad4a P 45a21a5f5f354af18836ebbe61a5a1ef: Deleting consensus metadata
I20260619 04:50:24.158799 23884 ts_tablet_manager.cc:1918] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:24.158843 23882 ts_tablet_manager.cc:1918] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:24.159027 23884 log.cc:1196] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/73d3db734b8d4a7db6076d2e4ab2ad4a
I20260619 04:50:24.159345 23370 catalog_manager.cc:4841] TS 45a21a5f5f354af18836ebbe61a5a1ef (127.3.243.129:39947): tablet 73d3db734b8d4a7db6076d2e4ab2ad4a (table test-table [id=e28f5d996909406586c402e76cb4b299]) successfully deleted
I20260619 04:50:24.159587 23884 ts_tablet_manager.cc:1939] T 73d3db734b8d4a7db6076d2e4ab2ad4a P eafd6e81c59c48d990666b02344b9b48: Deleting consensus metadata
I20260619 04:50:24.159718 23882 log.cc:1196] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/73d3db734b8d4a7db6076d2e4ab2ad4a
I20260619 04:50:24.160295 23882 ts_tablet_manager.cc:1939] T 73d3db734b8d4a7db6076d2e4ab2ad4a P c8a9bed721d24057b471dd9629a03818: Deleting consensus metadata
I20260619 04:50:24.161270 23372 catalog_manager.cc:4841] TS eafd6e81c59c48d990666b02344b9b48 (127.3.243.130:33695): tablet 73d3db734b8d4a7db6076d2e4ab2ad4a (table test-table [id=e28f5d996909406586c402e76cb4b299]) successfully deleted
I20260619 04:50:24.162015 23371 catalog_manager.cc:4841] TS c8a9bed721d24057b471dd9629a03818 (127.3.243.131:34785): tablet 73d3db734b8d4a7db6076d2e4ab2ad4a (table test-table [id=e28f5d996909406586c402e76cb4b299]) successfully deleted
Jun 19 04:50:24 dist-test-slave-2qp8 krb5kdc[23340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.129@KRBTEST.COM
I20260619 04:50:24.173166 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:24.139070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.3.243.129:39947)
0619 04:50:24.139325 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:24.139334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:24.154880 (+ 15546us) server_negotiation.cc:407] Connection header received
0619 04:50:24.155421 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:24.155426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:24.155515 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:24.155598 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:24.158611 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:24.159766 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:24.162174 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:24.162719 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:24.162868 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:24.167323 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:24.167335 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:24.167349 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:24.167381 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:24.170277 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:24.171044 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:24.171052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:24.171054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:24.171181 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:24.171569 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:24.171578 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:24.171580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:24.171924 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:24.172258 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:24.172590 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 04:50:24.172806 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 04:50:24.173830 23569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:24.268759 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 23446
I20260619 04:50:24.292292 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 23580
Jun 19 04:50:24 dist-test-slave-2qp8 krb5kdc[23340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.130@KRBTEST.COM
I20260619 04:50:24.308960 23858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:24.293058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.3.243.130:33695)
0619 04:50:24.293224 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:24.293229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:24.293313 (+ 84us) server_negotiation.cc:407] Connection header received
0619 04:50:24.293495 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:24.293498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:24.293563 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:24.293649 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:24.295186 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:24.296061 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:24.297606 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:24.298340 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:24.298458 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:24.303052 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:24.303065 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:24.303081 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:24.303117 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:24.306178 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:24.306826 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:24.306834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:24.306836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:24.306965 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:24.307372 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:24.307378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:24.307381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:24.307728 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:24.308105 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:24.308347 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 04:50:24.308565 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:50:24.309656 23703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:24.420324 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 23580
I20260619 04:50:24.438338 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 23713
Jun 19 04:50:24 dist-test-slave-2qp8 krb5kdc[23340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.131@KRBTEST.COM
I20260619 04:50:24.454908 23856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:24.439087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.3.243.131:34785)
0619 04:50:24.439365 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:24.439372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:24.439394 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:50:24.439593 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:24.439606 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:24.439676 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:24.439801 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:24.441229 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:24.441953 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:24.443435 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:24.443871 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:24.444037 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:24.449024 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:24.449039 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:24.449055 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:24.449100 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:24.451978 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:24.452685 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:24.452701 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:24.452704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:24.452864 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:24.453279 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:24.453289 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:24.453291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:24.453649 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:24.453957 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:24.454328 (+ 371us) server_negotiation.cc:299] Negotiation successful
0619 04:50:24.454559 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260619 04:50:24.455682 23836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:24.544530 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 23713
I20260619 04:50:24.561232 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 23353
I20260619 04:50:24.562379 23415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:24.673106 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 23353
2026-06-19T04:50:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4337 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T04:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:24Z Disabled control of system clock
I20260619 04:50:24.748934 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:40641
--webserver_interface=127.3.243.190
--webserver_port=0
--builtin_ntp_servers=127.3.243.148:42563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:40641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:50:25.070880 23893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:25.071182 23893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:25.071272 23893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:25.078286 23893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:25.078387 23893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:25.078466 23893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:25.078567 23893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:25.078629 23893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:25.087606 23893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:40641
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:40641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:25.088888 23893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:25.090643 23893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:25.102932 23899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:25.102906 23901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:25.103117 23898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:25.103760 23893 server_base.cc:1034] running on GCE node
I20260619 04:50:25.104660 23893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:25.106006 23893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:25.107255 23893 hybrid_clock.cc:648] HybridClock initialized: now 1781844625107203 us; error 48 us; skew 500 ppm
I20260619 04:50:25.107612 23893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:25.109910 23893 webserver.cc:460] Webserver started at http://127.3.243.190:44153/ using document root <none> and password file <none>
I20260619 04:50:25.110654 23893 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:25.110785 23893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:25.111104 23893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:25.114003 23893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "c3581383080d41bd8d3a45eede6d50a2"
format_stamp: "Formatted at 2026-06-19 04:50:25 on dist-test-slave-2qp8"
server_key: "ae67ae223815deaf0bb48230bc9bbe9a"
server_key_iv: "30bf01369f022fd3e3bd14c0fc6b106c"
server_key_version: "encryptionkey@0"
I20260619 04:50:25.114780 23893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "c3581383080d41bd8d3a45eede6d50a2"
format_stamp: "Formatted at 2026-06-19 04:50:25 on dist-test-slave-2qp8"
server_key: "ae67ae223815deaf0bb48230bc9bbe9a"
server_key_iv: "30bf01369f022fd3e3bd14c0fc6b106c"
server_key_version: "encryptionkey@0"
I20260619 04:50:25.119971 23893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:50:25.123508 23907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:25.124878 23893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:25.125065 23893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "c3581383080d41bd8d3a45eede6d50a2"
format_stamp: "Formatted at 2026-06-19 04:50:25 on dist-test-slave-2qp8"
server_key: "ae67ae223815deaf0bb48230bc9bbe9a"
server_key_iv: "30bf01369f022fd3e3bd14c0fc6b106c"
server_key_version: "encryptionkey@0"
I20260619 04:50:25.125280 23893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:25.140591 23893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:25.141423 23893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:25.141697 23893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:25.159634 23893 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:40641
I20260619 04:50:25.159657 23958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:40641 every 8 connection(s)
I20260619 04:50:25.161033 23893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:25.164763 23959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:25.171491 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 23893
I20260619 04:50:25.171661 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:25.172248 4046 external_mini_cluster.cc:1428] Setting key 844d8408123ff485219ea81a96b194b0
I20260619 04:50:25.175663 23959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2: Bootstrap starting.
I20260619 04:50:25.178737 23959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:25.179720 23959 log.cc:824] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:25.182461 23959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2: No bootstrap required, opened a new log
I20260619 04:50:25.185468 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:25.174140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.3.243.190:40641)
0619 04:50:25.175039 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:25.175050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:25.175090 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:50:25.176913 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:25.176939 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:25.177382 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:25.177751 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:25.178299 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:25.179593 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:25.181112 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:25.181876 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:25.182071 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:25.183165 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:25.183172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:25.183185 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:25.183705 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:25.183777 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:25.183804 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:25.183979 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:25.184226 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 04:50:25.184524 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":295,"threads_started":1}
I20260619 04:50:25.190112 23959 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3581383080d41bd8d3a45eede6d50a2" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40641 } }
I20260619 04:50:25.190511 23959 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:25.190615 23959 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3581383080d41bd8d3a45eede6d50a2, State: Initialized, Role: FOLLOWER
I20260619 04:50:25.191174 23959 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3581383080d41bd8d3a45eede6d50a2" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40641 } }
I20260619 04:50:25.191401 23959 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:25.191541 23959 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:25.191705 23959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:25.194051 23959 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3581383080d41bd8d3a45eede6d50a2" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40641 } }
I20260619 04:50:25.194546 23959 leader_election.cc:302] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3581383080d41bd8d3a45eede6d50a2; no voters:
I20260619 04:50:25.195125 23959 leader_election.cc:288] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:25.195277 23964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:25.196648 23964 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [term 1 LEADER]: Becoming Leader. State: Replica: c3581383080d41bd8d3a45eede6d50a2, State: Running, Role: LEADER
I20260619 04:50:25.197154 23964 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3581383080d41bd8d3a45eede6d50a2" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40641 } }
I20260619 04:50:25.197257 23959 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:25.199755 23966 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3581383080d41bd8d3a45eede6d50a2. Latest consensus state: current_term: 1 leader_uuid: "c3581383080d41bd8d3a45eede6d50a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3581383080d41bd8d3a45eede6d50a2" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40641 } } }
I20260619 04:50:25.200294 23965 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3581383080d41bd8d3a45eede6d50a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3581383080d41bd8d3a45eede6d50a2" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 40641 } } }
I20260619 04:50:25.200485 23966 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:25.200555 23965 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:25.204800 23973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:25.208892 23973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:25.217468 23973 catalog_manager.cc:1347] Generated new cluster ID: 5261e44eceec4debb45ec889154a8b9a
I20260619 04:50:25.217579 23973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:25.265295 23973 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:25.266641 23973 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:25.272782 23973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2: Generated new TSK 0
I20260619 04:50:25.273561 23973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:25.285068 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.129:0
--local_ip_for_outbound_sockets=127.3.243.129
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40641
--builtin_ntp_servers=127.3.243.148:42563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 04:50:25.624819 23983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:25.625128 23983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:25.625331 23983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:25.633037 23983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:25.633153 23983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:25.633348 23983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.129
I20260619 04:50:25.642863 23983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.243.129
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.129
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:25.644270 23983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:25.646040 23983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:25.658933 23988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:25.658975 23989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:25.658895 23991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:25.659626 23983 server_base.cc:1034] running on GCE node
I20260619 04:50:25.660204 23983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:25.661159 23983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:25.662520 23983 hybrid_clock.cc:648] HybridClock initialized: now 1781844625662460 us; error 46 us; skew 500 ppm
I20260619 04:50:25.662824 23983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:25.665182 23983 webserver.cc:460] Webserver started at http://127.3.243.129:38021/ using document root <none> and password file <none>
I20260619 04:50:25.665808 23983 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:25.665891 23983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:25.666147 23983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:25.668911 23983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/data/instance:
uuid: "274c940cb37343669cc6f46ebb614495"
format_stamp: "Formatted at 2026-06-19 04:50:25 on dist-test-slave-2qp8"
server_key: "8356914227ae2782e4b41e80aec5130d"
server_key_iv: "f86ef6564f3039263498083fab2e8fd9"
server_key_version: "encryptionkey@0"
I20260619 04:50:25.669629 23983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance:
uuid: "274c940cb37343669cc6f46ebb614495"
format_stamp: "Formatted at 2026-06-19 04:50:25 on dist-test-slave-2qp8"
server_key: "8356914227ae2782e4b41e80aec5130d"
server_key_iv: "f86ef6564f3039263498083fab2e8fd9"
server_key_version: "encryptionkey@0"
I20260619 04:50:25.674307 23983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 04:50:25.677767 23997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:25.679158 23983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:50:25.679476 23983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/wal
uuid: "274c940cb37343669cc6f46ebb614495"
format_stamp: "Formatted at 2026-06-19 04:50:25 on dist-test-slave-2qp8"
server_key: "8356914227ae2782e4b41e80aec5130d"
server_key_iv: "f86ef6564f3039263498083fab2e8fd9"
server_key_version: "encryptionkey@0"
I20260619 04:50:25.679731 23983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:25.706265 23983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:25.707190 23983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:25.707530 23983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:25.708642 23983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:25.710204 23983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:25.710340 23983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:25.710486 23983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:25.710587 23983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:25.746140 23983 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.129:38031
I20260619 04:50:25.746192 24109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.129:38031 every 8 connection(s)
I20260619 04:50:25.747577 23983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/data/info.pb
I20260619 04:50:25.750268 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 23983
I20260619 04:50:25.750458 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/wal/instance
I20260619 04:50:25.751046 4046 external_mini_cluster.cc:1428] Setting key a97cbb680d840da8ce9e34aa84ef3927
I20260619 04:50:25.754007 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.130:0
--local_ip_for_outbound_sockets=127.3.243.130
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40641
--builtin_ntp_servers=127.3.243.148:42563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:50:25.762219 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:25.750328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.129:54827 (local address 127.3.243.190:40641)
0619 04:50:25.750837 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:25.750844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:25.752163 (+ 1319us) server_negotiation.cc:407] Connection header received
0619 04:50:25.753990 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:25.753999 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:25.754096 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:25.754251 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:25.755485 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:25.756840 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:25.758534 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:25.759368 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:25.759525 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:25.761107 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:25.761116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:25.761124 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:25.761152 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:25.761223 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:25.761254 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:25.761362 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:25.761607 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 04:50:25.761727 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":236,"threads_started":1}
I20260619 04:50:25.763149 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:25.750932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:40641 (local address 127.3.243.129:54827)
0619 04:50:25.751988 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:25.752055 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:25.753707 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:25.754589 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:25.754599 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:25.754706 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:25.755293 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:25.755312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:25.757017 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:25.757021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:25.758372 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:25.758386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:25.759601 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:25.759606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:25.759840 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:25.760766 (+ 926us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:50:25.760769 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:25.760809 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:50:25.760893 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:25.761387 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:25.761397 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:25.761403 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:50:25.761559 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 04:50:25.761877 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":369,"threads_started":1}
I20260619 04:50:25.764976 24110 heartbeater.cc:347] Connected to a master server at 127.3.243.190:40641
I20260619 04:50:25.765305 24110 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:25.766083 24110 heartbeater.cc:511] Master 127.3.243.190:40641 requested a full tablet report, sending...
I20260619 04:50:25.768229 23924 ts_manager.cc:194] Registered new tserver with Master: 274c940cb37343669cc6f46ebb614495 (127.3.243.129:38031)
I20260619 04:50:25.769711 23924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.129:54827
W20260619 04:50:26.085132 24114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:26.085458 24114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:26.085613 24114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:26.092788 24114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:26.092924 24114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:26.093101 24114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.130
I20260619 04:50:26.102412 24114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.243.130
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.130
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:26.103695 24114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:26.105516 24114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:26.118594 24120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:26.118646 24122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:26.118538 24119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:26.119390 24114 server_base.cc:1034] running on GCE node
I20260619 04:50:26.119966 24114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:26.120922 24114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:26.122152 24114 hybrid_clock.cc:648] HybridClock initialized: now 1781844626122119 us; error 44 us; skew 500 ppm
I20260619 04:50:26.122483 24114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:26.124857 24114 webserver.cc:460] Webserver started at http://127.3.243.130:45325/ using document root <none> and password file <none>
I20260619 04:50:26.125559 24114 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:26.125685 24114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:26.126005 24114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:26.128911 24114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/data/instance:
uuid: "7c0e9b33447444eaaf82ca367ce1bf45"
format_stamp: "Formatted at 2026-06-19 04:50:26 on dist-test-slave-2qp8"
server_key: "5153291ef9f824ab879221087a688149"
server_key_iv: "a20139e190d3b7a6222a3c0d126b5f29"
server_key_version: "encryptionkey@0"
I20260619 04:50:26.129714 24114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance:
uuid: "7c0e9b33447444eaaf82ca367ce1bf45"
format_stamp: "Formatted at 2026-06-19 04:50:26 on dist-test-slave-2qp8"
server_key: "5153291ef9f824ab879221087a688149"
server_key_iv: "a20139e190d3b7a6222a3c0d126b5f29"
server_key_version: "encryptionkey@0"
I20260619 04:50:26.134552 24114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 04:50:26.137856 24128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:26.139088 24114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 04:50:26.139250 24114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/wal
uuid: "7c0e9b33447444eaaf82ca367ce1bf45"
format_stamp: "Formatted at 2026-06-19 04:50:26 on dist-test-slave-2qp8"
server_key: "5153291ef9f824ab879221087a688149"
server_key_iv: "a20139e190d3b7a6222a3c0d126b5f29"
server_key_version: "encryptionkey@0"
I20260619 04:50:26.139469 24114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:26.160380 24114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:26.161163 24114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:26.161489 24114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:26.162407 24114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:26.163888 24114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:26.164026 24114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:26.164160 24114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:26.164248 24114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:26.208175 24114 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.130:45961
I20260619 04:50:26.208284 24240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.130:45961 every 8 connection(s)
I20260619 04:50:26.209514 24114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/data/info.pb
I20260619 04:50:26.212337 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 24114
I20260619 04:50:26.212517 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/wal/instance
I20260619 04:50:26.212988 4046 external_mini_cluster.cc:1428] Setting key 7b790334d3d20e81adb80b225042ab63
I20260619 04:50:26.215770 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.243.131:0
--local_ip_for_outbound_sockets=127.3.243.131
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40641
--builtin_ntp_servers=127.3.243.148:42563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 04:50:26.223278 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.212206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.130:48807 (local address 127.3.243.190:40641)
0619 04:50:26.212438 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:26.212446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:26.214074 (+ 1628us) server_negotiation.cc:407] Connection header received
0619 04:50:26.215910 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:26.215918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:26.216014 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:26.216154 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:26.217254 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.218449 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.220037 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.220793 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.220936 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.222351 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:26.222359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:26.222366 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:26.222395 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:26.222439 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:26.222457 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:26.222550 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:26.222757 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 04:50:26.222881 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:50:26.224416 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.212788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:40641 (local address 127.3.243.130:48807)
0619 04:50:26.213899 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:26.213951 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:26.215638 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:26.216405 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:26.216413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:26.216516 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:26.217076 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.217093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.218609 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.218613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.219888 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.219900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.221011 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.221016 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.221219 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.222019 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:50:26.222022 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:26.222064 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:50:26.222153 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:26.222553 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:26.222561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:26.222567 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 04:50:26.222711 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 04:50:26.223033 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":366,"threads_started":1}
I20260619 04:50:26.225961 24241 heartbeater.cc:347] Connected to a master server at 127.3.243.190:40641
I20260619 04:50:26.226441 24241 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:26.227275 24241 heartbeater.cc:511] Master 127.3.243.190:40641 requested a full tablet report, sending...
I20260619 04:50:26.228858 23924 ts_manager.cc:194] Registered new tserver with Master: 7c0e9b33447444eaaf82ca367ce1bf45 (127.3.243.130:45961)
I20260619 04:50:26.229802 23924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.130:48807
W20260619 04:50:26.552704 24244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:26.552965 24244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:26.553097 24244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:26.560153 24244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:26.560246 24244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:26.560415 24244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.243.131
I20260619 04:50:26.569350 24244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.243.131
--webserver_port=0
--tserver_master_addrs=127.3.243.190:40641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.243.131
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:26.570595 24244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:26.572485 24244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:26.585649 24250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:26.585732 24249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:26.585759 24252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:26.586638 24244 server_base.cc:1034] running on GCE node
I20260619 04:50:26.587185 24244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:26.588059 24244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:26.589301 24244 hybrid_clock.cc:648] HybridClock initialized: now 1781844626589253 us; error 46 us; skew 500 ppm
I20260619 04:50:26.589644 24244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 04:50:26.591717 24244 webserver.cc:460] Webserver started at http://127.3.243.131:35947/ using document root <none> and password file <none>
I20260619 04:50:26.592471 24244 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:26.592597 24244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:26.592908 24244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:26.595577 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/data/instance:
uuid: "01e93827552e4d25982f9bdecec64f3c"
format_stamp: "Formatted at 2026-06-19 04:50:26 on dist-test-slave-2qp8"
server_key: "029f27b840174e905732053273dbeeed"
server_key_iv: "33e5037e09064aa2e498a672902cc661"
server_key_version: "encryptionkey@0"
I20260619 04:50:26.596422 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance:
uuid: "01e93827552e4d25982f9bdecec64f3c"
format_stamp: "Formatted at 2026-06-19 04:50:26 on dist-test-slave-2qp8"
server_key: "029f27b840174e905732053273dbeeed"
server_key_iv: "33e5037e09064aa2e498a672902cc661"
server_key_version: "encryptionkey@0"
I20260619 04:50:26.601634 24244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:50:26.605130 24258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:26.606437 24244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 04:50:26.606606 24244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/wal
uuid: "01e93827552e4d25982f9bdecec64f3c"
format_stamp: "Formatted at 2026-06-19 04:50:26 on dist-test-slave-2qp8"
server_key: "029f27b840174e905732053273dbeeed"
server_key_iv: "33e5037e09064aa2e498a672902cc661"
server_key_version: "encryptionkey@0"
I20260619 04:50:26.606834 24244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:26.630079 24244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:26.630915 24244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:26.631259 24244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:26.632285 24244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 04:50:26.633759 24244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 04:50:26.633898 24244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:26.634022 24244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 04:50:26.634114 24244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:26.667179 24244 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.131:36451
I20260619 04:50:26.667280 24370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.131:36451 every 8 connection(s)
I20260619 04:50:26.668495 24244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/data/info.pb
I20260619 04:50:26.673609 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 24244
I20260619 04:50:26.673815 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/wal/instance
I20260619 04:50:26.674506 4046 external_mini_cluster.cc:1428] Setting key 28b50d926a3d64ba7d182f1859f1c4c7
I20260619 04:50:26.683128 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.671299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:46553 (local address 127.3.243.190:40641)
0619 04:50:26.671524 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:26.671530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:26.673064 (+ 1534us) server_negotiation.cc:407] Connection header received
0619 04:50:26.675749 (+ 2685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:26.675754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:26.675844 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:26.675973 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:26.677285 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.678137 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.679914 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.680493 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.680677 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.682132 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:26.682141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:26.682152 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:26.682178 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:26.682211 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:26.682224 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:26.682330 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:26.682584 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 04:50:26.682706 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:50:26.684417 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.671879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:40641 (local address 127.3.243.131:46553)
0619 04:50:26.672894 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:26.672940 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:26.675494 (+ 2554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:26.676291 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:26.676304 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:26.676440 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:50:26.677088 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.677107 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.678332 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.678344 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.679745 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.679759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.680806 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.680814 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.681055 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.681806 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 04:50:26.681809 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:50:26.681846 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 04:50:26.681930 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:50:26.682327 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:50:26.682356 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:50:26.682367 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 04:50:26.682573 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 04:50:26.683023 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":359,"threads_started":1}
I20260619 04:50:26.685757 24371 heartbeater.cc:347] Connected to a master server at 127.3.243.190:40641
I20260619 04:50:26.686069 24371 heartbeater.cc:464] Registering TS with master...
I20260619 04:50:26.686877 24371 heartbeater.cc:511] Master 127.3.243.190:40641 requested a full tablet report, sending...
I20260619 04:50:26.688382 23924 ts_manager.cc:194] Registered new tserver with Master: 01e93827552e4d25982f9bdecec64f3c (127.3.243.131:36451)
I20260619 04:50:26.689424 23924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.243.131:46553
I20260619 04:50:26.689841 4046 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 04:50:26.698956 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.693739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.3.243.190:40641)
0619 04:50:26.693964 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:26.693973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:26.694320 (+ 347us) server_negotiation.cc:407] Connection header received
0619 04:50:26.694420 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:26.694423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:26.694489 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:26.694567 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:26.694979 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.695614 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.696921 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.697402 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.697502 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.698420 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:26.698424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:26.698427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:26.698442 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:26.698470 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:26.698481 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:26.698549 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:26.698666 (+ 117us) server_negotiation.cc:299] Negotiation successful
0619 04:50:26.698728 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 04:50:26.702900 23924 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 04:50:26.710307 23924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 04:50:26.742213 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.722795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.3.243.131:36451)
0619 04:50:26.734871 (+ 12076us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:26.734877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:26.734922 (+ 45us) server_negotiation.cc:407] Connection header received
0619 04:50:26.734997 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:26.735002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:26.735206 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:26.735413 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:26.736184 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.737690 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.739643 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.741039 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.741187 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.741464 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:26.741677 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 04:50:26.741842 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11910,"thread_start_us":11758,"threads_started":1}
I20260619 04:50:26.742545 24383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.721582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:45961 (local address 127.0.0.1:54512)
0619 04:50:26.722502 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:26.722549 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:26.722759 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:26.734801 (+ 12042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:26.734808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:26.734845 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:26.735301 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.735316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.737335 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.737340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.739785 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.739800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.741746 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.741752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.741930 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.741963 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:50:26.742028 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 04:50:26.742173 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":4,"thread_start_us":362,"threads_started":1}
I20260619 04:50:26.742237 24386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.722235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.131:36451 (local address 127.0.0.1:59728)
0619 04:50:26.732925 (+ 10690us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:26.732953 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:26.733136 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:26.735485 (+ 2349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:26.735489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:26.735513 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:26.735975 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.735986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.737838 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.737844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.739478 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.739489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.741218 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.741223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.741423 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.741480 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 04:50:26.741628 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 04:50:26.741820 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10470,"mutex_wait_us":101,"thread_start_us":11274,"threads_started":1}
I20260619 04:50:26.743185 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.721923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.3.243.130:45961)
0619 04:50:26.734071 (+ 12148us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:26.734095 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:26.734131 (+ 36us) server_negotiation.cc:407] Connection header received
0619 04:50:26.734252 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:26.734260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:26.734540 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:26.735508 (+ 968us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:26.735736 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.737185 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.740015 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.741521 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.742342 (+ 821us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.742552 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:26.742624 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 04:50:26.742810 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11999,"thread_start_us":11903,"threads_started":1}
I20260619 04:50:26.743578 24385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.722723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:38031 (local address 127.0.0.1:60702)
0619 04:50:26.733195 (+ 10472us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:26.733229 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:26.733389 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:26.735185 (+ 1796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:26.735190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:26.735212 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:26.735738 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.735751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.737706 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.737711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.739961 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.739975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.742690 (+ 2715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.742697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.742934 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.742968 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 04:50:26.743057 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 04:50:26.743262 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10304,"mutex_wait_us":77,"thread_start_us":10528,"threads_started":1}
I20260619 04:50:26.743645 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.723313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.3.243.129:38031)
0619 04:50:26.734533 (+ 11220us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:26.734540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:26.734569 (+ 29us) server_negotiation.cc:407] Connection header received
0619 04:50:26.734667 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:26.734677 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:26.734969 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:26.735191 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:26.735931 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.737566 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.740707 (+ 3141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.742146 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.742301 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.742552 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:26.743104 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 04:50:26.743294 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11057,"thread_start_us":10861,"threads_started":1}
I20260619 04:50:26.747083 24176 tablet_service.cc:1467] Processing CreateTablet for tablet 3edee5c7700343829aac331ea44a39be (DEFAULT_TABLE table=test-table [id=6c2c39bcbb014496861b107899b5bdd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:26.747079 24306 tablet_service.cc:1467] Processing CreateTablet for tablet 3edee5c7700343829aac331ea44a39be (DEFAULT_TABLE table=test-table [id=6c2c39bcbb014496861b107899b5bdd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:26.747226 24045 tablet_service.cc:1467] Processing CreateTablet for tablet 3edee5c7700343829aac331ea44a39be (DEFAULT_TABLE table=test-table [id=6c2c39bcbb014496861b107899b5bdd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 04:50:26.748503 24306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3edee5c7700343829aac331ea44a39be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:26.748498 24045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3edee5c7700343829aac331ea44a39be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:26.748517 24176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3edee5c7700343829aac331ea44a39be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:26.761499 24389 tablet_bootstrap.cc:492] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: Bootstrap starting.
I20260619 04:50:26.763371 24390 tablet_bootstrap.cc:492] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: Bootstrap starting.
I20260619 04:50:26.764163 24391 tablet_bootstrap.cc:492] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: Bootstrap starting.
I20260619 04:50:26.766495 24389 tablet_bootstrap.cc:654] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:26.766860 24390 tablet_bootstrap.cc:654] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:26.767717 24391 tablet_bootstrap.cc:654] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:26.767750 24389 log.cc:824] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:26.767745 24390 log.cc:824] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:26.768894 24391 log.cc:824] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:26.771013 24390 tablet_bootstrap.cc:492] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: No bootstrap required, opened a new log
I20260619 04:50:26.771014 24389 tablet_bootstrap.cc:492] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: No bootstrap required, opened a new log
I20260619 04:50:26.771252 24390 ts_tablet_manager.cc:1397] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260619 04:50:26.771268 24389 ts_tablet_manager.cc:1397] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 04:50:26.771736 24391 tablet_bootstrap.cc:492] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: No bootstrap required, opened a new log
I20260619 04:50:26.771973 24391 ts_tablet_manager.cc:1397] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 04:50:26.772795 24110 heartbeater.cc:503] Master 127.3.243.190:40641 was elected leader, sending a full tablet report...
I20260619 04:50:26.780273 24390 raft_consensus.cc:348] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } }
I20260619 04:50:26.780695 24390 raft_consensus.cc:374] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:26.780833 24390 raft_consensus.cc:729] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e93827552e4d25982f9bdecec64f3c, State: Initialized, Role: FOLLOWER
I20260619 04:50:26.781456 24390 consensus_queue.cc:260] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } }
I20260619 04:50:26.781836 24391 raft_consensus.cc:348] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } }
I20260619 04:50:26.782521 24371 heartbeater.cc:503] Master 127.3.243.190:40641 was elected leader, sending a full tablet report...
I20260619 04:50:26.782369 24389 raft_consensus.cc:348] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } }
I20260619 04:50:26.782773 24389 raft_consensus.cc:374] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:26.782878 24389 raft_consensus.cc:729] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0e9b33447444eaaf82ca367ce1bf45, State: Initialized, Role: FOLLOWER
I20260619 04:50:26.783336 24390 ts_tablet_manager.cc:1428] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20260619 04:50:26.782318 24391 raft_consensus.cc:374] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:26.783501 24389 consensus_queue.cc:260] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } }
I20260619 04:50:26.783730 24391 raft_consensus.cc:729] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 274c940cb37343669cc6f46ebb614495, State: Initialized, Role: FOLLOWER
I20260619 04:50:26.784401 24391 consensus_queue.cc:260] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } }
I20260619 04:50:26.784765 24241 heartbeater.cc:503] Master 127.3.243.190:40641 was elected leader, sending a full tablet report...
I20260619 04:50:26.785462 24389 ts_tablet_manager.cc:1428] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260619 04:50:26.787220 24391 ts_tablet_manager.cc:1428] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260619 04:50:26.920445 24372 tablet.cc:2369] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:26.938200 24395 raft_consensus.cc:482] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 04:50:26.938524 24395 raft_consensus.cc:504] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } }
I20260619 04:50:26.940232 24395 leader_election.cc:288] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c0e9b33447444eaaf82ca367ce1bf45 (127.3.243.130:45961), 274c940cb37343669cc6f46ebb614495 (127.3.243.129:38031)
I20260619 04:50:26.947098 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.940441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.130:45961 (local address 127.3.243.131:39097)
0619 04:50:26.940634 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:26.940655 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:26.940801 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:26.941323 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:26.941327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:26.941353 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:26.941521 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.941535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.942868 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.942873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.944629 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.944640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.946352 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.946358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.946565 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.946598 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 04:50:26.946684 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 04:50:26.946818 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":29}
I20260619 04:50:26.947608 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.940697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:39097 (local address 127.3.243.130:45961)
0619 04:50:26.940903 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:26.940910 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:26.940928 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:50:26.941036 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:26.941043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:26.941132 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:26.941227 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:26.941703 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.942699 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.944782 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.946030 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.946176 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.946267 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:26.946954 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 04:50:26.947086 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:50:26.947901 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.940804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.129:38031 (local address 127.3.243.131:34301)
0619 04:50:26.941440 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0619 04:50:26.941460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:50:26.941553 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:50:26.941859 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:50:26.941864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:50:26.941886 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 04:50:26.942258 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.942270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.943687 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.943690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.945566 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:50:26.945581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.947246 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.947251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:50:26.947412 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.947432 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 04:50:26.947501 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 04:50:26.947575 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":18,"thread_start_us":292,"threads_started":1}
I20260619 04:50:26.947964 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.940962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.243.131:34301 (local address 127.3.243.129:38031)
0619 04:50:26.941177 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:26.941183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:26.941551 (+ 368us) server_negotiation.cc:407] Connection header received
0619 04:50:26.941660 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:26.941667 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:26.941753 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:26.941821 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 04:50:26.942409 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.943491 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.945754 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.947079 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.947222 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.947278 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:26.947572 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 04:50:26.947710 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 04:50:26.948559 24196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3edee5c7700343829aac331ea44a39be" candidate_uuid: "01e93827552e4d25982f9bdecec64f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" is_pre_election: true
I20260619 04:50:26.948963 24065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3edee5c7700343829aac331ea44a39be" candidate_uuid: "01e93827552e4d25982f9bdecec64f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274c940cb37343669cc6f46ebb614495" is_pre_election: true
I20260619 04:50:26.949152 24196 raft_consensus.cc:2454] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e93827552e4d25982f9bdecec64f3c in term 0.
I20260619 04:50:26.949515 24065 raft_consensus.cc:2454] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e93827552e4d25982f9bdecec64f3c in term 0.
I20260619 04:50:26.949818 24260 leader_election.cc:302] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e93827552e4d25982f9bdecec64f3c, 7c0e9b33447444eaaf82ca367ce1bf45; no voters:
I20260619 04:50:26.950186 24395 raft_consensus.cc:2781] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 04:50:26.950325 24395 raft_consensus.cc:482] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 04:50:26.950425 24395 raft_consensus.cc:3037] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:26.952620 24395 raft_consensus.cc:504] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } }
I20260619 04:50:26.953161 24395 leader_election.cc:288] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [CANDIDATE]: Term 1 election: Requested vote from peers 7c0e9b33447444eaaf82ca367ce1bf45 (127.3.243.130:45961), 274c940cb37343669cc6f46ebb614495 (127.3.243.129:38031)
I20260619 04:50:26.953542 24196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3edee5c7700343829aac331ea44a39be" candidate_uuid: "01e93827552e4d25982f9bdecec64f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c0e9b33447444eaaf82ca367ce1bf45"
I20260619 04:50:26.953644 24065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3edee5c7700343829aac331ea44a39be" candidate_uuid: "01e93827552e4d25982f9bdecec64f3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "274c940cb37343669cc6f46ebb614495"
I20260619 04:50:26.953789 24196 raft_consensus.cc:3037] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:26.953862 24065 raft_consensus.cc:3037] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:26.957048 24065 raft_consensus.cc:2454] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e93827552e4d25982f9bdecec64f3c in term 1.
I20260619 04:50:26.957055 24196 raft_consensus.cc:2454] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e93827552e4d25982f9bdecec64f3c in term 1.
I20260619 04:50:26.957432 24260 leader_election.cc:302] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e93827552e4d25982f9bdecec64f3c, 274c940cb37343669cc6f46ebb614495; no voters:
I20260619 04:50:26.957695 24395 raft_consensus.cc:2781] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:26.958123 24395 raft_consensus.cc:686] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 1 LEADER]: Becoming Leader. State: Replica: 01e93827552e4d25982f9bdecec64f3c, State: Running, Role: LEADER
I20260619 04:50:26.958577 24395 consensus_queue.cc:237] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } }
W20260619 04:50:26.961498 24242 tablet.cc:2369] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:26.963331 23924 catalog_manager.cc:5495] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 01e93827552e4d25982f9bdecec64f3c (127.3.243.131). New cstate: current_term: 1 leader_uuid: "01e93827552e4d25982f9bdecec64f3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01e93827552e4d25982f9bdecec64f3c" member_type: VOTER last_known_addr { host: "127.3.243.131" port: 36451 } health_report { overall_health: HEALTHY } } }
I20260619 04:50:26.988850 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:26.982489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.3.243.131:36451)
0619 04:50:26.982785 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:26.982791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:26.982846 (+ 55us) server_negotiation.cc:407] Connection header received
0619 04:50:26.983031 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:26.983037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:26.983208 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:26.983386 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:50:26.984074 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.984915 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.986330 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:26.986724 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:26.986819 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:26.987278 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:50:26.987784 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:50:26.987926 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:26.988461 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 04:50:26.988641 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 04:50:26.997150 24065 raft_consensus.cc:1261] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [term 1 FOLLOWER]: Refusing update from remote peer 01e93827552e4d25982f9bdecec64f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:26.997193 24196 raft_consensus.cc:1261] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [term 1 FOLLOWER]: Refusing update from remote peer 01e93827552e4d25982f9bdecec64f3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:50:26.998158 24395 consensus_queue.cc:1035] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c0e9b33447444eaaf82ca367ce1bf45" member_type: VOTER last_known_addr { host: "127.3.243.130" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:50:26.998497 24399 consensus_queue.cc:1035] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "274c940cb37343669cc6f46ebb614495" member_type: VOTER last_known_addr { host: "127.3.243.129" port: 38031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 04:50:27.000478 24111 tablet.cc:2369] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:50:27.017078 24403 mvcc.cc:205] Tried to move back new op lower bound from 7298435592170344448 to 7298435592026075136. Current Snapshot: MvccSnapshot[applied={T|T < 7298435592170344448}]
I20260619 04:50:27.023447 23923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:27.023763 23923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 04:50:27.027498 23923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3581383080d41bd8d3a45eede6d50a2: Sending DeleteTablet for 3 replicas of tablet 3edee5c7700343829aac331ea44a39be
I20260619 04:50:27.028661 24176 tablet_service.cc:1514] Processing DeleteTablet for tablet 3edee5c7700343829aac331ea44a39be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:27 UTC) from {username='slave'} at 127.0.0.1:54512
I20260619 04:50:27.028659 24306 tablet_service.cc:1514] Processing DeleteTablet for tablet 3edee5c7700343829aac331ea44a39be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:27 UTC) from {username='slave'} at 127.0.0.1:59728
I20260619 04:50:27.029254 24045 tablet_service.cc:1514] Processing DeleteTablet for tablet 3edee5c7700343829aac331ea44a39be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 04:50:27 UTC) from {username='slave'} at 127.0.0.1:60702
I20260619 04:50:27.030678 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 23983
I20260619 04:50:27.040997 24414 tablet_replica.cc:330] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: stopping tablet replica
I20260619 04:50:27.041512 24414 raft_consensus.cc:2229] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:27.042078 24414 raft_consensus.cc:2258] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:27.046274 24414 ts_tablet_manager.cc:1905] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:27.054814 24414 ts_tablet_manager.cc:1918] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:27.054996 24414 log.cc:1196] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-0/wal/wals/3edee5c7700343829aac331ea44a39be
I20260619 04:50:27.055548 24414 ts_tablet_manager.cc:1939] T 3edee5c7700343829aac331ea44a39be P 274c940cb37343669cc6f46ebb614495: Deleting consensus metadata
I20260619 04:50:27.057986 23909 catalog_manager.cc:4841] TS 274c940cb37343669cc6f46ebb614495 (127.3.243.129:38031): tablet 3edee5c7700343829aac331ea44a39be (table test-table [id=6c2c39bcbb014496861b107899b5bdd6]) successfully deleted
I20260619 04:50:27.058218 24413 tablet_replica.cc:330] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: stopping tablet replica
I20260619 04:50:27.058665 24413 raft_consensus.cc:2229] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 04:50:27.059173 24413 raft_consensus.cc:2258] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:27.060001 24412 tablet_replica.cc:330] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: stopping tablet replica
I20260619 04:50:27.060427 24412 raft_consensus.cc:2229] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 1 LEADER]: Raft consensus shutting down.
I20260619 04:50:27.061060 24412 raft_consensus.cc:2258] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 04:50:27.061694 24413 ts_tablet_manager.cc:1905] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:27.062728 24412 ts_tablet_manager.cc:1905] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 04:50:27.067251 24388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:27.032281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60716 (local address 127.3.243.129:38031)
0619 04:50:27.032465 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:27.032472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:27.055803 (+ 23331us) server_negotiation.cc:407] Connection header received
0619 04:50:27.056343 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:27.056348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:27.056466 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:27.056552 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:27.057178 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:27.058296 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:27.060192 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:27.060777 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:27.060867 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:27.065559 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:27.065570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:27.065587 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:27.066357 (+ 770us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:27.066419 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:27.066444 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:27.066525 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:27.066768 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 04:50:27.066879 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 04:50:27.068629 24105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:27.068672 24413 ts_tablet_manager.cc:1918] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:27.068831 24413 log.cc:1196] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-1/wal/wals/3edee5c7700343829aac331ea44a39be
I20260619 04:50:27.069020 24412 ts_tablet_manager.cc:1918] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 04:50:27.069181 24412 log.cc:1196] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: Deleting WAL directory at /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781844312210132-4046-0/minicluster-data/ts-2/wal/wals/3edee5c7700343829aac331ea44a39be
I20260619 04:50:27.069487 24413 ts_tablet_manager.cc:1939] T 3edee5c7700343829aac331ea44a39be P 7c0e9b33447444eaaf82ca367ce1bf45: Deleting consensus metadata
I20260619 04:50:27.069839 24412 ts_tablet_manager.cc:1939] T 3edee5c7700343829aac331ea44a39be P 01e93827552e4d25982f9bdecec64f3c: Deleting consensus metadata
I20260619 04:50:27.071504 23911 catalog_manager.cc:4841] TS 01e93827552e4d25982f9bdecec64f3c (127.3.243.131:36451): tablet 3edee5c7700343829aac331ea44a39be (table test-table [id=6c2c39bcbb014496861b107899b5bdd6]) successfully deleted
I20260619 04:50:27.071650 23909 catalog_manager.cc:4841] TS 7c0e9b33447444eaaf82ca367ce1bf45 (127.3.243.130:45961): tablet 3edee5c7700343829aac331ea44a39be (table test-table [id=6c2c39bcbb014496861b107899b5bdd6]) successfully deleted
I20260619 04:50:27.161370 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 23983
I20260619 04:50:27.184649 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24114
I20260619 04:50:27.192090 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:27.185441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54524 (local address 127.3.243.130:45961)
0619 04:50:27.185608 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:27.185613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:27.185695 (+ 82us) server_negotiation.cc:407] Connection header received
0619 04:50:27.185876 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:27.185880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:27.185975 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:27.186064 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:27.186562 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:27.187288 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:27.189066 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:27.189444 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:27.189535 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:27.190666 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:27.190675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:27.190688 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:27.191283 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:27.191336 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:27.191355 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:27.191466 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:27.191647 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 04:50:27.191732 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:50:27.192806 24236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:27.281689 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 24114
I20260619 04:50:27.299928 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24244
I20260619 04:50:27.307358 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:27.300905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.3.243.131:36451)
0619 04:50:27.301100 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:27.301106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:27.301127 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:50:27.301272 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:27.301276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:27.301444 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:27.301599 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:27.302017 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:27.302844 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:27.304137 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:27.304595 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:27.304740 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:27.305860 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:27.305869 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:27.305883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:27.306517 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:27.306574 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:27.306593 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:27.306721 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:27.306878 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 04:50:27.306979 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 04:50:27.308549 24366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:27.432642 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 24244
I20260619 04:50:27.452227 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 23893
I20260619 04:50:27.453291 23954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:27.561010 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 23893
2026-06-19T04:50:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2887 ms)
[----------] 6 tests from EncryptionPolicyTest (22230 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:27 dist-test-slave-2qp8 krb5kdc[24422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:27 dist-test-slave-2qp8 krb5kdc[24422](info): set up 2 sockets
Jun 19 04:50:27 dist-test-slave-2qp8 krb5kdc[24422](info): commencing operation
krb5kdc: starting...
W20260619 04:50:28.668766 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:28 dist-test-slave-2qp8 krb5kdc[24422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:28.913599 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34295
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:29.236415 24435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:29.236713 24435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:29.236897 24435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:29.243993 24435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:29.244100 24435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:29.244211 24435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:29.244324 24435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:29.244410 24435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:29.253894 24435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:29.255123 24435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:29.256390 24435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:29.268018 24443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:29.268018 24441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:29.268227 24440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:29.268980 24435 server_base.cc:1034] running on GCE node
I20260619 04:50:29.269722 24435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:29.271119 24435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:29.272344 24435 hybrid_clock.cc:648] HybridClock initialized: now 1781844629272301 us; error 54 us; skew 500 ppm
I20260619 04:50:29.272747 24435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:29 dist-test-slave-2qp8 krb5kdc[24422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:29.275949 24435 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:29.277418 24435 webserver.cc:460] Webserver started at http://127.0.0.1:35515/ using document root <none> and password file <none>
I20260619 04:50:29.278105 24435 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:29.278242 24435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:29.278532 24435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:29.281590 24435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "cc8a1487a91f4f3794e965f0851de51a"
format_stamp: "Formatted at 2026-06-19 04:50:29 on dist-test-slave-2qp8"
server_key: "d43c8f12327ee998f335b8dfc12b3d7e"
server_key_iv: "ea5afb209a462ec1d6be2b3266914690"
server_key_version: "encryptionkey@0"
I20260619 04:50:29.282338 24435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "cc8a1487a91f4f3794e965f0851de51a"
format_stamp: "Formatted at 2026-06-19 04:50:29 on dist-test-slave-2qp8"
server_key: "d43c8f12327ee998f335b8dfc12b3d7e"
server_key_iv: "ea5afb209a462ec1d6be2b3266914690"
server_key_version: "encryptionkey@0"
I20260619 04:50:29.287405 24435 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:50:29.290885 24450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:29.292171 24435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:50:29.292358 24435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "cc8a1487a91f4f3794e965f0851de51a"
format_stamp: "Formatted at 2026-06-19 04:50:29 on dist-test-slave-2qp8"
server_key: "d43c8f12327ee998f335b8dfc12b3d7e"
server_key_iv: "ea5afb209a462ec1d6be2b3266914690"
server_key_version: "encryptionkey@0"
I20260619 04:50:29.292544 24435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:29.303954 24435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:29.308396 24435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:29.308668 24435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:29.326920 24435 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34295
I20260619 04:50:29.326967 24501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34295 every 8 connection(s)
I20260619 04:50:29.328296 24435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:29.332053 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:29.337190 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 24435
I20260619 04:50:29.337400 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:29.338037 4046 external_mini_cluster.cc:1428] Setting key fe16a5381854c3b2d91f92f5eb011754
I20260619 04:50:29.343709 24502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a: Bootstrap starting.
I20260619 04:50:29.347455 24502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:29.348659 24502 log.cc:824] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:29.351542 24502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a: No bootstrap required, opened a new log
Jun 19 04:50:29 dist-test-slave-2qp8 krb5kdc[24422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 04:50:29.362198 24502 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8a1487a91f4f3794e965f0851de51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } }
I20260619 04:50:29.362629 24502 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:29.362746 24502 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8a1487a91f4f3794e965f0851de51a, State: Initialized, Role: FOLLOWER
I20260619 04:50:29.363236 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:29.340066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50993 (local address 127.0.0.1:34295)
0619 04:50:29.340969 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:29.340989 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:29.341047 (+ 58us) server_negotiation.cc:407] Connection header received
0619 04:50:29.343455 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:29.343483 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:29.344000 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:29.344459 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:29.345813 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:29.347791 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:29.349252 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:29.350070 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:29.350253 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:29.354588 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:29.354600 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:29.354620 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:29.354663 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:29.358708 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:29.359431 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:29.359442 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:29.359456 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:29.359675 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:29.360078 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:29.360086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:29.360090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:29.360854 (+ 764us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:29.361172 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:29.361554 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 04:50:29.361889 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":379,"threads_started":1}
I20260619 04:50:29.363417 24502 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8a1487a91f4f3794e965f0851de51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } }
I20260619 04:50:29.363698 24502 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:29.363853 24502 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:29.364099 24502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:29.366809 24502 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8a1487a91f4f3794e965f0851de51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } }
I20260619 04:50:29.367347 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:29.367373 24502 leader_election.cc:302] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc8a1487a91f4f3794e965f0851de51a; no voters:
I20260619 04:50:29.368057 24502 leader_election.cc:288] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:29.368196 24507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:29.370605 24502 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:29.373929 24507 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [term 1 LEADER]: Becoming Leader. State: Replica: cc8a1487a91f4f3794e965f0851de51a, State: Running, Role: LEADER
I20260619 04:50:29.374675 24507 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8a1487a91f4f3794e965f0851de51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } }
I20260619 04:50:29.378127 24509 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc8a1487a91f4f3794e965f0851de51a. Latest consensus state: current_term: 1 leader_uuid: "cc8a1487a91f4f3794e965f0851de51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8a1487a91f4f3794e965f0851de51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } } }
I20260619 04:50:29.378571 24509 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:29.377815 24508 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc8a1487a91f4f3794e965f0851de51a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8a1487a91f4f3794e965f0851de51a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34295 } } }
I20260619 04:50:29.379169 24508 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:29.387046 24522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:29.392791 24522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:29.398826 24505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:29.377421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54185 (local address 127.0.0.1:34295)
0619 04:50:29.377595 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:29.377601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:29.379246 (+ 1645us) server_negotiation.cc:407] Connection header received
0619 04:50:29.379506 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:29.379516 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:29.379622 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:29.379758 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:29.382468 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:29.383986 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:29.386338 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:29.387055 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:29.387168 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:29.390968 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:29.390975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:29.390980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:29.391024 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:29.395053 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:29.396129 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:29.396140 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:29.396144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:29.396287 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:29.396999 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:29.397004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:29.397006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:29.397406 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:29.397622 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:29.398268 (+ 646us) server_negotiation.cc:299] Negotiation successful
0619 04:50:29.398494 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 04:50:29.403657 24522 catalog_manager.cc:1347] Generated new cluster ID: 69f540b6f2ce4c20be95fbc86ea2fdb2
I20260619 04:50:29.403770 24522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:29.417578 24522 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:29.419333 24522 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:29.436913 24522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc8a1487a91f4f3794e965f0851de51a: Generated new TSK 0
I20260619 04:50:29.438011 24522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:29.450109 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24435
I20260619 04:50:29.452277 24497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:29.543172 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 24435
W20260619 04:50:29.560669 24420 connection.cc:535] client connection to 127.0.0.1:34295 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34295): Connection reset by peer (error 104)
2026-06-19T04:50:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:29 dist-test-slave-2qp8 krb5kdc[24535](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:29 dist-test-slave-2qp8 krb5kdc[24535](info): set up 2 sockets
Jun 19 04:50:29 dist-test-slave-2qp8 krb5kdc[24535](info): commencing operation
krb5kdc: starting...
W20260619 04:50:30.656375 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:30 dist-test-slave-2qp8 krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:30.898854 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40655
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:31.222651 24548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:31.222980 24548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:31.223138 24548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:31.230211 24548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:31.230379 24548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:31.230472 24548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:31.230564 24548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:31.230630 24548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:31.240087 24548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:31.241436 24548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:31.242776 24548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:31.254303 24554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:31.254211 24553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:31.254354 24556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:31.255369 24548 server_base.cc:1034] running on GCE node
I20260619 04:50:31.256039 24548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:31.257404 24548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:31.258633 24548 hybrid_clock.cc:648] HybridClock initialized: now 1781844631258595 us; error 49 us; skew 500 ppm
I20260619 04:50:31.258975 24548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:31 dist-test-slave-2qp8 krb5kdc[24535](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:31.262128 24548 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:31.263656 24548 webserver.cc:460] Webserver started at http://127.0.0.1:45475/ using document root <none> and password file <none>
I20260619 04:50:31.264429 24548 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:31.264575 24548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:31.264873 24548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:31.267724 24548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "0907fde311694573a456c446903b2e11"
format_stamp: "Formatted at 2026-06-19 04:50:31 on dist-test-slave-2qp8"
server_key: "6738434960f87590753826161ad7b4dc"
server_key_iv: "cbbfa242ed432ee1dff53709b2fc12e8"
server_key_version: "encryptionkey@0"
I20260619 04:50:31.268553 24548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "0907fde311694573a456c446903b2e11"
format_stamp: "Formatted at 2026-06-19 04:50:31 on dist-test-slave-2qp8"
server_key: "6738434960f87590753826161ad7b4dc"
server_key_iv: "cbbfa242ed432ee1dff53709b2fc12e8"
server_key_version: "encryptionkey@0"
I20260619 04:50:31.273437 24548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 04:50:31.276955 24563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:31.278203 24548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:50:31.278406 24548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "0907fde311694573a456c446903b2e11"
format_stamp: "Formatted at 2026-06-19 04:50:31 on dist-test-slave-2qp8"
server_key: "6738434960f87590753826161ad7b4dc"
server_key_iv: "cbbfa242ed432ee1dff53709b2fc12e8"
server_key_version: "encryptionkey@0"
I20260619 04:50:31.278597 24548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:31.293205 24548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:31.297547 24548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:31.297832 24548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:31.316128 24548 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40655
I20260619 04:50:31.316220 24614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40655 every 8 connection(s)
I20260619 04:50:31.317523 24548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:31.321211 24615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:31.322640 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 24548
I20260619 04:50:31.322831 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:31.323282 4046 external_mini_cluster.cc:1428] Setting key 4d1269634ad25fba5f120c3c30fd9ef6
I20260619 04:50:31.335554 24615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11: Bootstrap starting.
I20260619 04:50:31.337811 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:31.325198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54539 (local address 127.0.0.1:40655)
0619 04:50:31.325973 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:31.325987 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:31.326039 (+ 52us) server_negotiation.cc:407] Connection header received
0619 04:50:31.327799 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:31.327834 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:31.328380 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:31.328844 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:31.329317 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:31.331146 (+ 1829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:31.332513 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:31.333613 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:31.333783 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:31.335072 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:31.335083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:31.335104 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:31.335788 (+ 684us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:31.335894 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:31.335929 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:31.336092 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:31.336380 (+ 288us) server_negotiation.cc:299] Negotiation successful
0619 04:50:31.336532 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":315,"threads_started":1}
I20260619 04:50:31.339259 24615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:31.340408 24615 log.cc:824] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:31.341630 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:31.343029 24615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11: No bootstrap required, opened a new log
I20260619 04:50:31.352444 24618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:31.346046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36803 (local address 127.0.0.1:40655)
0619 04:50:31.346224 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:31.346230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:31.346521 (+ 291us) server_negotiation.cc:407] Connection header received
0619 04:50:31.346660 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:31.346665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:31.346742 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:31.346858 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:31.347289 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:31.348524 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:31.349765 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:31.350464 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:31.350582 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:31.351595 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:31.351601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:31.351605 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:31.351624 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:31.351686 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:31.351709 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:31.351823 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:31.351986 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 04:50:31.352033 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:50:31.352137 24615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0907fde311694573a456c446903b2e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260619 04:50:31.352715 24615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:31.352859 24615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0907fde311694573a456c446903b2e11, State: Initialized, Role: FOLLOWER
I20260619 04:50:31.353475 24615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0907fde311694573a456c446903b2e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260619 04:50:31.353727 24615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:31.353896 24615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:31.354068 24615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:31.356225 24615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0907fde311694573a456c446903b2e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260619 04:50:31.356720 24615 leader_election.cc:302] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0907fde311694573a456c446903b2e11; no voters:
I20260619 04:50:31.357282 24615 leader_election.cc:288] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:31.357409 24625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:31.358204 24625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [term 1 LEADER]: Becoming Leader. State: Replica: 0907fde311694573a456c446903b2e11, State: Running, Role: LEADER
I20260619 04:50:31.358848 24625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0907fde311694573a456c446903b2e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } }
I20260619 04:50:31.359627 24615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:31.361686 24626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0907fde311694573a456c446903b2e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0907fde311694573a456c446903b2e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } } }
I20260619 04:50:31.361714 24627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0907fde311694573a456c446903b2e11. Latest consensus state: current_term: 1 leader_uuid: "0907fde311694573a456c446903b2e11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0907fde311694573a456c446903b2e11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40655 } } }
I20260619 04:50:31.362172 24627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:31.362172 24626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:31.364351 24630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:31.369009 24630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:31.377691 24630 catalog_manager.cc:1347] Generated new cluster ID: 5f4df65e0270475b913c545c80baea78
I20260619 04:50:31.377802 24630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:31.400336 24630 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:31.401798 24630 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:31.410164 24630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0907fde311694573a456c446903b2e11: Generated new TSK 0
I20260619 04:50:31.411204 24630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:31.425370 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24548
W20260619 04:50:31.426421 24610 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54539
W20260619 04:50:31.427256 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24548: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:31.427435 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 24548
W20260619 04:50:31.444621 24533 connection.cc:535] client connection to 127.0.0.1:40655 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40655): Connection reset by peer (error 104)
2026-06-19T04:50:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1884 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:31 dist-test-slave-2qp8 krb5kdc[24646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:31 dist-test-slave-2qp8 krb5kdc[24646](info): set up 2 sockets
Jun 19 04:50:31 dist-test-slave-2qp8 krb5kdc[24646](info): commencing operation
krb5kdc: starting...
W20260619 04:50:32.528879 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.009s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:32 dist-test-slave-2qp8 krb5kdc[24646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:32.761458 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45619
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:33.083608 24659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:33.083868 24659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:33.084081 24659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:33.090994 24659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:33.091097 24659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:33.091225 24659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:33.091332 24659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:33.091414 24659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:33.101702 24659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45619
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:33.103051 24659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:33.104436 24659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:33.116436 24667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:33.116478 24665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:33.116391 24664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:33.117511 24659 server_base.cc:1034] running on GCE node
I20260619 04:50:33.118324 24659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:33.119709 24659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:33.120908 24659 hybrid_clock.cc:648] HybridClock initialized: now 1781844633120868 us; error 31 us; skew 500 ppm
I20260619 04:50:33.121214 24659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:33 dist-test-slave-2qp8 krb5kdc[24646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:33.128305 24659 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:33.129714 24659 webserver.cc:460] Webserver started at http://127.0.0.1:33911/ using document root <none> and password file <none>
I20260619 04:50:33.130425 24659 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:33.130553 24659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:33.130870 24659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:33.133944 24659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "eb08346e735c4910b62bfc56cab37bf5"
format_stamp: "Formatted at 2026-06-19 04:50:33 on dist-test-slave-2qp8"
server_key: "a762738a489c8888b51e5a2376312f0b"
server_key_iv: "78f1038ba6d87019506477ea0bbe018b"
server_key_version: "encryptionkey@0"
I20260619 04:50:33.134688 24659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "eb08346e735c4910b62bfc56cab37bf5"
format_stamp: "Formatted at 2026-06-19 04:50:33 on dist-test-slave-2qp8"
server_key: "a762738a489c8888b51e5a2376312f0b"
server_key_iv: "78f1038ba6d87019506477ea0bbe018b"
server_key_version: "encryptionkey@0"
I20260619 04:50:33.139722 24659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:50:33.143059 24674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:33.144539 24659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:33.144721 24659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "eb08346e735c4910b62bfc56cab37bf5"
format_stamp: "Formatted at 2026-06-19 04:50:33 on dist-test-slave-2qp8"
server_key: "a762738a489c8888b51e5a2376312f0b"
server_key_iv: "78f1038ba6d87019506477ea0bbe018b"
server_key_version: "encryptionkey@0"
I20260619 04:50:33.144918 24659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:33.150637 24659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:33.154654 24659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:33.154958 24659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:33.173235 24659 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45619
I20260619 04:50:33.173389 24725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45619 every 8 connection(s)
I20260619 04:50:33.174685 24659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:33.178376 24726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:33.183714 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 24659
I20260619 04:50:33.183903 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:33.184448 4046 external_mini_cluster.cc:1428] Setting key 8d4859a062b6a2a29f3470095c1b0521
I20260619 04:50:33.189638 24726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5: Bootstrap starting.
I20260619 04:50:33.191869 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:33.186193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46399 (local address 127.0.0.1:45619)
0619 04:50:33.187023 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:33.187033 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:33.187075 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:50:33.188775 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:33.188796 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:33.189127 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:33.189431 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:33.190396 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:33.190402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:33.190414 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:33.190425 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:33.190479 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:33.190504 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:33.190662 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:33.190777 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 04:50:33.190959 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":377,"threads_started":1}
I20260619 04:50:33.193217 24726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:33.194296 24726 log.cc:824] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:33.195372 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:33.197103 24726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5: No bootstrap required, opened a new log
I20260619 04:50:33.202406 24729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:33.199752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49485 (local address 127.0.0.1:45619)
0619 04:50:33.199953 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:33.199961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:33.200481 (+ 520us) server_negotiation.cc:407] Connection header received
0619 04:50:33.200633 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:33.200636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:33.200678 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:33.200751 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:33.201725 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:33.201728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:33.201731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:33.201745 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:33.201795 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:33.201810 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:33.201900 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:33.202037 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 04:50:33.202082 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 04:50:33.205392 24726 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08346e735c4910b62bfc56cab37bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45619 } }
I20260619 04:50:33.205749 24726 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:33.205864 24726 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb08346e735c4910b62bfc56cab37bf5, State: Initialized, Role: FOLLOWER
I20260619 04:50:33.206416 24726 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08346e735c4910b62bfc56cab37bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45619 } }
I20260619 04:50:33.206643 24726 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:33.206784 24726 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:33.206952 24726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:33.209143 24726 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08346e735c4910b62bfc56cab37bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45619 } }
I20260619 04:50:33.209602 24726 leader_election.cc:302] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb08346e735c4910b62bfc56cab37bf5; no voters:
I20260619 04:50:33.210117 24726 leader_election.cc:288] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:33.210287 24736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:33.210989 24736 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [term 1 LEADER]: Becoming Leader. State: Replica: eb08346e735c4910b62bfc56cab37bf5, State: Running, Role: LEADER
I20260619 04:50:33.211525 24736 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08346e735c4910b62bfc56cab37bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45619 } }
I20260619 04:50:33.212167 24726 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:33.214258 24738 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb08346e735c4910b62bfc56cab37bf5. Latest consensus state: current_term: 1 leader_uuid: "eb08346e735c4910b62bfc56cab37bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08346e735c4910b62bfc56cab37bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45619 } } }
I20260619 04:50:33.214715 24738 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:33.216652 24745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:33.216605 24737 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb08346e735c4910b62bfc56cab37bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08346e735c4910b62bfc56cab37bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45619 } } }
I20260619 04:50:33.216988 24737 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:33.221510 24745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:33.229625 24745 catalog_manager.cc:1347] Generated new cluster ID: da0591fc00fa4b0684387f8d22586031
I20260619 04:50:33.229730 24745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:33.241874 24745 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:33.242262 24745 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:33.252537 24745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb08346e735c4910b62bfc56cab37bf5: Generated new TSK 0
I20260619 04:50:33.253294 24745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:33.260342 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24659
W20260619 04:50:33.261409 24721 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46399
W20260619 04:50:33.262145 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24659: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:33.262300 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 24659
2026-06-19T04:50:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:33 dist-test-slave-2qp8 krb5kdc[24757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:33 dist-test-slave-2qp8 krb5kdc[24757](info): set up 2 sockets
Jun 19 04:50:33 dist-test-slave-2qp8 krb5kdc[24757](info): commencing operation
krb5kdc: starting...
W20260619 04:50:34.376844 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:34 dist-test-slave-2qp8 krb5kdc[24757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:34.619438 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:34.935523 24770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:34.935853 24770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:34.936022 24770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:34.943029 24770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:34.943179 24770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:34.943301 24770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:34.943410 24770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:34.943498 24770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:34.952817 24770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:34.954042 24770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:34.955289 24770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:34.966740 24775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:34.966789 24778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:34.967056 24776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:34.967752 24770 server_base.cc:1034] running on GCE node
I20260619 04:50:34.968556 24770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:34.969945 24770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:34.971163 24770 hybrid_clock.cc:648] HybridClock initialized: now 1781844634971135 us; error 43 us; skew 500 ppm
I20260619 04:50:34.971496 24770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:34 dist-test-slave-2qp8 krb5kdc[24757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:34.974577 24770 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:34.976001 24770 webserver.cc:460] Webserver started at http://127.0.0.1:45195/ using document root <none> and password file <none>
I20260619 04:50:34.976718 24770 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:34.976863 24770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:34.977197 24770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:34.980048 24770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "e9866d4ff2914fddaa8fffc1ca7874d4"
format_stamp: "Formatted at 2026-06-19 04:50:34 on dist-test-slave-2qp8"
server_key: "06ece790880b0e47ff637bc494a199d3"
server_key_iv: "9cfce4c7be2ffd733e0d82dd0515abbd"
server_key_version: "encryptionkey@0"
I20260619 04:50:34.980852 24770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "e9866d4ff2914fddaa8fffc1ca7874d4"
format_stamp: "Formatted at 2026-06-19 04:50:34 on dist-test-slave-2qp8"
server_key: "06ece790880b0e47ff637bc494a199d3"
server_key_iv: "9cfce4c7be2ffd733e0d82dd0515abbd"
server_key_version: "encryptionkey@0"
I20260619 04:50:34.985792 24770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:50:34.989074 24785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:34.990247 24770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 04:50:34.990438 24770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "e9866d4ff2914fddaa8fffc1ca7874d4"
format_stamp: "Formatted at 2026-06-19 04:50:34 on dist-test-slave-2qp8"
server_key: "06ece790880b0e47ff637bc494a199d3"
server_key_iv: "9cfce4c7be2ffd733e0d82dd0515abbd"
server_key_version: "encryptionkey@0"
I20260619 04:50:34.990622 24770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:35.022275 24770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:35.026944 24770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:35.027258 24770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:35.045677 24770 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41401
I20260619 04:50:35.045698 24836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41401 every 8 connection(s)
I20260619 04:50:35.047111 24770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:35.050755 24837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:35.052804 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 24770
I20260619 04:50:35.052985 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:35.053431 4046 external_mini_cluster.cc:1428] Setting key 2cc6cdbaa221246dd54951eebe8bb3f9
I20260619 04:50:35.063643 24837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4: Bootstrap starting.
I20260619 04:50:35.067014 24837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:35.067967 24837 log.cc:824] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4: Log is configured to *not* fsync() on all Append() calls
Jun 19 04:50:35 dist-test-slave-2qp8 krb5kdc[24757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 04:50:35.071274 24837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4: No bootstrap required, opened a new log
I20260619 04:50:35.077833 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:35.055230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50871 (local address 127.0.0.1:41401)
0619 04:50:35.056042 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:35.056057 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:35.056131 (+ 74us) server_negotiation.cc:407] Connection header received
0619 04:50:35.058480 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:35.058506 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:35.059005 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:35.059420 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:35.060754 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:35.062775 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:35.064137 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:35.065127 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:35.065332 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:35.069619 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:35.069629 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:35.069645 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:35.069687 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:35.073554 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:35.074160 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:35.074168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:35.074176 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:35.074388 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:35.074771 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:35.074776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:35.074781 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:35.075472 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:35.075770 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:35.076047 (+ 277us) server_negotiation.cc:299] Negotiation successful
0619 04:50:35.076487 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":333,"threads_started":1}
I20260619 04:50:35.081542 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:35.081692 24837 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9866d4ff2914fddaa8fffc1ca7874d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41401 } }
I20260619 04:50:35.082048 24837 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:35.082150 24837 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9866d4ff2914fddaa8fffc1ca7874d4, State: Initialized, Role: FOLLOWER
I20260619 04:50:35.082733 24837 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9866d4ff2914fddaa8fffc1ca7874d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41401 } }
I20260619 04:50:35.082983 24837 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:35.083115 24837 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:35.083292 24837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:35.085608 24837 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9866d4ff2914fddaa8fffc1ca7874d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41401 } }
I20260619 04:50:35.086159 24837 leader_election.cc:302] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9866d4ff2914fddaa8fffc1ca7874d4; no voters:
I20260619 04:50:35.086845 24837 leader_election.cc:288] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:35.086969 24847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:35.087656 24847 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [term 1 LEADER]: Becoming Leader. State: Replica: e9866d4ff2914fddaa8fffc1ca7874d4, State: Running, Role: LEADER
I20260619 04:50:35.088418 24847 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9866d4ff2914fddaa8fffc1ca7874d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41401 } }
I20260619 04:50:35.090646 24847 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9866d4ff2914fddaa8fffc1ca7874d4. Latest consensus state: current_term: 1 leader_uuid: "e9866d4ff2914fddaa8fffc1ca7874d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9866d4ff2914fddaa8fffc1ca7874d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41401 } } }
I20260619 04:50:35.091073 24847 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:35.091383 24849 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9866d4ff2914fddaa8fffc1ca7874d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9866d4ff2914fddaa8fffc1ca7874d4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41401 } } }
I20260619 04:50:35.091806 24849 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:35.095681 24837 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:35.097390 24851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:35.102787 24851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:35.103906 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:35.085935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40939 (local address 127.0.0.1:41401)
0619 04:50:35.086147 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:35.086154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:35.086650 (+ 496us) server_negotiation.cc:407] Connection header received
0619 04:50:35.086840 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:35.086845 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:35.086942 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:35.087049 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:35.088803 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:35.089597 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:35.091285 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:35.092050 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:35.092168 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:35.095803 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:35.095812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:35.095820 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:35.095878 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:35.099901 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:35.100793 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:35.100799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:35.100802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:35.100947 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:35.101439 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:35.101444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:35.101447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:35.101863 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:35.102084 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:35.103272 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0619 04:50:35.103503 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:50:35.114939 24851 catalog_manager.cc:1347] Generated new cluster ID: d83f9a604249405c8aea2d4e73a6e92e
I20260619 04:50:35.115054 24851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:35.139017 24851 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:35.140996 24851 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:35.153219 24851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9866d4ff2914fddaa8fffc1ca7874d4: Generated new TSK 0
I20260619 04:50:35.154214 24851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:35.161208 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24770
I20260619 04:50:35.162757 24832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:35.252913 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 24770
2026-06-19T04:50:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:35 dist-test-slave-2qp8 krb5kdc[24869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:35 dist-test-slave-2qp8 krb5kdc[24869](info): set up 2 sockets
Jun 19 04:50:35 dist-test-slave-2qp8 krb5kdc[24869](info): commencing operation
krb5kdc: starting...
W20260619 04:50:36.352717 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:36 dist-test-slave-2qp8 krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:36.603665 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:36.928954 24882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:36.929293 24882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:36.929451 24882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:36.936769 24882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:36.936913 24882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:36.937016 24882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:36.937120 24882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:36.937208 24882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:36.946857 24882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:36.948173 24882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:36.949545 24882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:36.961542 24887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:36.961500 24890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:36.961580 24888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:36.962373 24882 server_base.cc:1034] running on GCE node
I20260619 04:50:36.963094 24882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:36.964486 24882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:36.965724 24882 hybrid_clock.cc:648] HybridClock initialized: now 1781844636965675 us; error 64 us; skew 500 ppm
I20260619 04:50:36.966075 24882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:36 dist-test-slave-2qp8 krb5kdc[24869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:36.969059 24882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:36.970321 24882 webserver.cc:460] Webserver started at http://127.0.0.1:37461/ using document root <none> and password file <none>
I20260619 04:50:36.970973 24882 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:36.971071 24882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:36.971309 24882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:36.974195 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "abf64149d6644d4f9589735d196dd43e"
format_stamp: "Formatted at 2026-06-19 04:50:36 on dist-test-slave-2qp8"
server_key: "65322d87dba83c4477e1f00fbdfe01a4"
server_key_iv: "7377c8132f7860ca8b28b9b23b28c56a"
server_key_version: "encryptionkey@0"
I20260619 04:50:36.974889 24882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "abf64149d6644d4f9589735d196dd43e"
format_stamp: "Formatted at 2026-06-19 04:50:36 on dist-test-slave-2qp8"
server_key: "65322d87dba83c4477e1f00fbdfe01a4"
server_key_iv: "7377c8132f7860ca8b28b9b23b28c56a"
server_key_version: "encryptionkey@0"
I20260619 04:50:36.980057 24882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 04:50:36.983297 24897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:36.984656 24882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:36.984839 24882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "abf64149d6644d4f9589735d196dd43e"
format_stamp: "Formatted at 2026-06-19 04:50:36 on dist-test-slave-2qp8"
server_key: "65322d87dba83c4477e1f00fbdfe01a4"
server_key_iv: "7377c8132f7860ca8b28b9b23b28c56a"
server_key_version: "encryptionkey@0"
I20260619 04:50:36.985026 24882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:37.017413 24882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:37.022596 24882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:37.022912 24882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:37.040921 24882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36349
I20260619 04:50:37.040956 24948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36349 every 8 connection(s)
I20260619 04:50:37.042286 24882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:37.045982 24949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:37.046892 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 24882
I20260619 04:50:37.047075 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:37.047660 4046 external_mini_cluster.cc:1428] Setting key 4f1807adf182166e5dcbda2597d42b8e
I20260619 04:50:37.058470 24949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e: Bootstrap starting.
I20260619 04:50:37.061218 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:37.049522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53673 (local address 127.0.0.1:36349)
0619 04:50:37.050249 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:37.050267 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:37.050305 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:50:37.052606 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:37.052628 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:37.053068 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:37.053486 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:37.054027 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:37.055338 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:37.056906 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:37.057551 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:37.057763 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:37.058986 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:37.058993 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:37.059006 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:37.059547 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:37.059620 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:37.059648 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:37.059803 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:37.060026 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 04:50:37.060299 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":335,"threads_started":1}
I20260619 04:50:37.061825 24949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:37.062991 24949 log.cc:824] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:37.065003 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:37.066160 24949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e: No bootstrap required, opened a new log
I20260619 04:50:37.076071 24949 raft_consensus.cc:348] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf64149d6644d4f9589735d196dd43e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36349 } }
I20260619 04:50:37.076630 24949 raft_consensus.cc:374] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:37.076754 24949 raft_consensus.cc:729] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf64149d6644d4f9589735d196dd43e, State: Initialized, Role: FOLLOWER
I20260619 04:50:37.076861 24952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:37.070071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60647 (local address 127.0.0.1:36349)
0619 04:50:37.070261 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:37.070266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:37.070742 (+ 476us) server_negotiation.cc:407] Connection header received
0619 04:50:37.070882 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:37.070887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:37.070958 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:37.071060 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:37.071579 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:37.072710 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:37.074014 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:37.074736 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:37.074863 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:37.075941 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:37.075947 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:37.075952 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:37.075971 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:37.076034 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:37.076061 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:37.076198 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:37.076398 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 04:50:37.076488 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:50:37.077380 24949 consensus_queue.cc:260] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf64149d6644d4f9589735d196dd43e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36349 } }
I20260619 04:50:37.077642 24949 raft_consensus.cc:388] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:37.077788 24949 raft_consensus.cc:482] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:37.077970 24949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:37.080251 24949 raft_consensus.cc:504] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf64149d6644d4f9589735d196dd43e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36349 } }
I20260619 04:50:37.080736 24949 leader_election.cc:302] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abf64149d6644d4f9589735d196dd43e; no voters:
I20260619 04:50:37.081317 24949 leader_election.cc:288] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:37.081456 24959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:37.082166 24959 raft_consensus.cc:686] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [term 1 LEADER]: Becoming Leader. State: Replica: abf64149d6644d4f9589735d196dd43e, State: Running, Role: LEADER
I20260619 04:50:37.082693 24959 consensus_queue.cc:237] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf64149d6644d4f9589735d196dd43e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36349 } }
I20260619 04:50:37.083498 24949 sys_catalog.cc:564] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:37.085455 24961 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [sys.catalog]: SysCatalogTable state changed. Reason: New leader abf64149d6644d4f9589735d196dd43e. Latest consensus state: current_term: 1 leader_uuid: "abf64149d6644d4f9589735d196dd43e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf64149d6644d4f9589735d196dd43e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36349 } } }
I20260619 04:50:37.085856 24961 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:37.087291 24960 sys_catalog.cc:455] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abf64149d6644d4f9589735d196dd43e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abf64149d6644d4f9589735d196dd43e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36349 } } }
I20260619 04:50:37.087872 24960 sys_catalog.cc:458] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:37.087981 24967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:37.092743 24967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:37.100564 24967 catalog_manager.cc:1347] Generated new cluster ID: e7becfb4873f488da432e00d9615e902
I20260619 04:50:37.100673 24967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:37.126698 24967 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:37.128676 24967 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:37.136332 24967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abf64149d6644d4f9589735d196dd43e: Generated new TSK 0
I20260619 04:50:37.137384 24967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:37.152051 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24882
W20260619 04:50:37.153257 24944 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53673
W20260619 04:50:37.153960 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24882: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:37.154107 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 24882
2026-06-19T04:50:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:37 dist-test-slave-2qp8 krb5kdc[24980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:37 dist-test-slave-2qp8 krb5kdc[24980](info): set up 2 sockets
Jun 19 04:50:37 dist-test-slave-2qp8 krb5kdc[24980](info): commencing operation
krb5kdc: starting...
W20260619 04:50:38.256641 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:38 dist-test-slave-2qp8 krb5kdc[24980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:38.503101 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:38.821098 24993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:38.821440 24993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:38.821605 24993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:38.828671 24993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:38.828804 24993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:38.828902 24993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:38.829001 24993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:38.829085 24993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:38.838613 24993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:38.839854 24993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:38.841074 24993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:38.854091 24998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:38.854126 25001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:38.854178 24999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:38.855022 24993 server_base.cc:1034] running on GCE node
I20260619 04:50:38.855805 24993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:38.857201 24993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:38.858428 24993 hybrid_clock.cc:648] HybridClock initialized: now 1781844638858388 us; error 50 us; skew 500 ppm
I20260619 04:50:38.858772 24993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:38 dist-test-slave-2qp8 krb5kdc[24980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:38.865315 24993 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:38.866832 24993 webserver.cc:460] Webserver started at http://127.0.0.1:33535/ using document root <none> and password file <none>
I20260619 04:50:38.867532 24993 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:38.867687 24993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:38.867998 24993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:38.871115 24993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "cf8f3fef5bfe432980210d20d8b8dc21"
format_stamp: "Formatted at 2026-06-19 04:50:38 on dist-test-slave-2qp8"
server_key: "307153b95252d2587002068dfc053621"
server_key_iv: "e217c4b60516803df424113adaaee554"
server_key_version: "encryptionkey@0"
I20260619 04:50:38.871917 24993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "cf8f3fef5bfe432980210d20d8b8dc21"
format_stamp: "Formatted at 2026-06-19 04:50:38 on dist-test-slave-2qp8"
server_key: "307153b95252d2587002068dfc053621"
server_key_iv: "e217c4b60516803df424113adaaee554"
server_key_version: "encryptionkey@0"
I20260619 04:50:38.877415 24993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:50:38.880934 25008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:38.882189 24993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:50:38.882376 24993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "cf8f3fef5bfe432980210d20d8b8dc21"
format_stamp: "Formatted at 2026-06-19 04:50:38 on dist-test-slave-2qp8"
server_key: "307153b95252d2587002068dfc053621"
server_key_iv: "e217c4b60516803df424113adaaee554"
server_key_version: "encryptionkey@0"
I20260619 04:50:38.882566 24993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:38.887863 24993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:38.891610 24993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:38.891885 24993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:38.908982 24993 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33381
I20260619 04:50:38.909026 25059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33381 every 8 connection(s)
I20260619 04:50:38.910312 24993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:38.913895 25060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:38.915019 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 24993
I20260619 04:50:38.915196 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:38.915751 4046 external_mini_cluster.cc:1428] Setting key 1a5b79937878f8725a282ca7d62f1c0b
I20260619 04:50:38.923242 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:38.917588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49015 (local address 127.0.0.1:33381)
0619 04:50:38.918331 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:38.918345 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:38.918383 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:50:38.920037 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:38.920057 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:38.920444 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:38.920817 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:38.921746 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:38.921752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:38.921765 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:38.921779 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:38.921834 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:38.921869 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:38.922101 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:38.922251 (+ 150us) server_negotiation.cc:299] Negotiation successful
0619 04:50:38.922416 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":315,"threads_started":1}
I20260619 04:50:38.926982 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:38.927191 25060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21: Bootstrap starting.
I20260619 04:50:38.930172 25060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:38.931432 25060 log.cc:824] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:38.933764 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:38.931208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38879 (local address 127.0.0.1:33381)
0619 04:50:38.931374 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:38.931386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:38.931783 (+ 397us) server_negotiation.cc:407] Connection header received
0619 04:50:38.931986 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:38.931995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:38.932056 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:38.932212 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:38.933143 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:38.933147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:38.933149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:38.933163 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:38.933197 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:38.933211 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:38.933297 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:38.933440 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 04:50:38.933483 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 04:50:38.934566 25060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21: No bootstrap required, opened a new log
I20260619 04:50:38.941689 25060 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f3fef5bfe432980210d20d8b8dc21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33381 } }
I20260619 04:50:38.942082 25060 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:38.942206 25060 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8f3fef5bfe432980210d20d8b8dc21, State: Initialized, Role: FOLLOWER
I20260619 04:50:38.942734 25060 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f3fef5bfe432980210d20d8b8dc21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33381 } }
I20260619 04:50:38.942970 25060 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:38.943099 25060 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:38.943293 25060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:38.945494 25060 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f3fef5bfe432980210d20d8b8dc21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33381 } }
I20260619 04:50:38.945910 25060 leader_election.cc:302] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf8f3fef5bfe432980210d20d8b8dc21; no voters:
I20260619 04:50:38.946416 25060 leader_election.cc:288] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:38.946545 25070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:38.947204 25070 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [term 1 LEADER]: Becoming Leader. State: Replica: cf8f3fef5bfe432980210d20d8b8dc21, State: Running, Role: LEADER
I20260619 04:50:38.947755 25070 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f3fef5bfe432980210d20d8b8dc21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33381 } }
I20260619 04:50:38.948458 25060 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:38.950707 25071 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf8f3fef5bfe432980210d20d8b8dc21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f3fef5bfe432980210d20d8b8dc21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33381 } } }
I20260619 04:50:38.951066 25071 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:38.950798 25072 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf8f3fef5bfe432980210d20d8b8dc21. Latest consensus state: current_term: 1 leader_uuid: "cf8f3fef5bfe432980210d20d8b8dc21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8f3fef5bfe432980210d20d8b8dc21" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33381 } } }
I20260619 04:50:38.951453 25072 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:38.960793 25076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:38.967253 25076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:38.975790 25076 catalog_manager.cc:1347] Generated new cluster ID: 91d426b0bb3f462fa6d8a07db89ae7bf
I20260619 04:50:38.975893 25076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:39.003878 25076 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:39.004338 25076 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:39.010667 25076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf8f3fef5bfe432980210d20d8b8dc21: Generated new TSK 0
I20260619 04:50:39.011387 25076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:39.023393 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24993
W20260619 04:50:39.024466 25055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49015
W20260619 04:50:39.025115 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 24993: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:39.025247 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 24993
2026-06-19T04:50:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1868 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:39 dist-test-slave-2qp8 krb5kdc[25091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:39 dist-test-slave-2qp8 krb5kdc[25091](info): set up 2 sockets
Jun 19 04:50:39 dist-test-slave-2qp8 krb5kdc[25091](info): commencing operation
krb5kdc: starting...
W20260619 04:50:40.176748 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:40 dist-test-slave-2qp8 krb5kdc[25091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:40.425626 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:40.745529 25104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:40.745807 25104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:40.745903 25104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:40.752902 25104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:40.753001 25104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:40.753046 25104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:40.753101 25104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:40.753142 25104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:40.762455 25104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:40.763655 25104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:40.764958 25104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:40.776090 25110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:40.776283 25112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:40.776312 25109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:40.777242 25104 server_base.cc:1034] running on GCE node
I20260619 04:50:40.777899 25104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:40.779109 25104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:40.780292 25104 hybrid_clock.cc:648] HybridClock initialized: now 1781844640780246 us; error 39 us; skew 500 ppm
I20260619 04:50:40.780632 25104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:40 dist-test-slave-2qp8 krb5kdc[25091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:40.783670 25104 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:40.785116 25104 webserver.cc:460] Webserver started at http://127.0.0.1:46089/ using document root <none> and password file <none>
I20260619 04:50:40.785781 25104 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:40.785876 25104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:40.786100 25104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:40.789011 25104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "ade07e2713fc49f2acec06e481b27603"
format_stamp: "Formatted at 2026-06-19 04:50:40 on dist-test-slave-2qp8"
server_key: "296ce6ac4aecd7e383178727b8c63958"
server_key_iv: "a478ce7e5cb8c887260e320edf947db8"
server_key_version: "encryptionkey@0"
I20260619 04:50:40.789695 25104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "ade07e2713fc49f2acec06e481b27603"
format_stamp: "Formatted at 2026-06-19 04:50:40 on dist-test-slave-2qp8"
server_key: "296ce6ac4aecd7e383178727b8c63958"
server_key_iv: "a478ce7e5cb8c887260e320edf947db8"
server_key_version: "encryptionkey@0"
I20260619 04:50:40.794653 25104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 04:50:40.797952 25119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:40.799222 25104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:50:40.799404 25104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "ade07e2713fc49f2acec06e481b27603"
format_stamp: "Formatted at 2026-06-19 04:50:40 on dist-test-slave-2qp8"
server_key: "296ce6ac4aecd7e383178727b8c63958"
server_key_iv: "a478ce7e5cb8c887260e320edf947db8"
server_key_version: "encryptionkey@0"
I20260619 04:50:40.799582 25104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:40.812677 25104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:40.816864 25104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:40.817137 25104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:40.835688 25104 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46157
I20260619 04:50:40.835740 25170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46157 every 8 connection(s)
I20260619 04:50:40.837065 25104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:40.838390 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 25104
I20260619 04:50:40.838582 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:40.839214 4046 external_mini_cluster.cc:1428] Setting key 0346cc8660c6fdc9a93dad0d92ec1372
I20260619 04:50:40.841822 25171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:50:40 dist-test-slave-2qp8 krb5kdc[25091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 04:50:40.855342 25171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603: Bootstrap starting.
I20260619 04:50:40.859853 25171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:40.861249 25171 log.cc:824] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:40.864068 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:40.841066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:42091 (local address 127.0.0.1:46157)
0619 04:50:40.841797 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:40.841808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:40.841857 (+ 49us) server_negotiation.cc:407] Connection header received
0619 04:50:40.843451 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:40.843471 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:40.843818 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:40.844220 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:40.846172 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:40.847424 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:40.849481 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:40.850108 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:40.850236 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:40.855242 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:40.855254 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:40.855272 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:40.855320 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:40.859328 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:40.860202 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:40.860211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:40.860221 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:40.860475 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:40.860940 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:40.860947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:40.860954 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:40.861667 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:40.862065 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:40.862765 (+ 700us) server_negotiation.cc:299] Negotiation successful
0619 04:50:40.863067 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":340,"threads_started":1}
I20260619 04:50:40.864414 25171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603: No bootstrap required, opened a new log
I20260619 04:50:40.867931 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:40.872561 25171 raft_consensus.cc:348] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade07e2713fc49f2acec06e481b27603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46157 } }
I20260619 04:50:40.873010 25171 raft_consensus.cc:374] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:40.873132 25171 raft_consensus.cc:729] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ade07e2713fc49f2acec06e481b27603, State: Initialized, Role: FOLLOWER
I20260619 04:50:40.873711 25171 consensus_queue.cc:260] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade07e2713fc49f2acec06e481b27603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46157 } }
I20260619 04:50:40.873929 25171 raft_consensus.cc:388] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:40.874078 25171 raft_consensus.cc:482] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:40.874254 25171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:40.876996 25171 raft_consensus.cc:504] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade07e2713fc49f2acec06e481b27603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46157 } }
I20260619 04:50:40.877560 25171 leader_election.cc:302] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ade07e2713fc49f2acec06e481b27603; no voters:
I20260619 04:50:40.878182 25171 leader_election.cc:288] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:40.878335 25181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:40.879300 25181 raft_consensus.cc:686] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [term 1 LEADER]: Becoming Leader. State: Replica: ade07e2713fc49f2acec06e481b27603, State: Running, Role: LEADER
I20260619 04:50:40.879968 25181 consensus_queue.cc:237] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade07e2713fc49f2acec06e481b27603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46157 } }
I20260619 04:50:40.880618 25171 sys_catalog.cc:564] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:40.882174 25182 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ade07e2713fc49f2acec06e481b27603. Latest consensus state: current_term: 1 leader_uuid: "ade07e2713fc49f2acec06e481b27603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade07e2713fc49f2acec06e481b27603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46157 } } }
I20260619 04:50:40.882575 25182 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:40.884245 25183 sys_catalog.cc:455] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ade07e2713fc49f2acec06e481b27603" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ade07e2713fc49f2acec06e481b27603" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46157 } } }
I20260619 04:50:40.884639 25183 sys_catalog.cc:458] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:40.886822 25187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:40.890377 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:40.872225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:37753 (local address 127.0.0.1:46157)
0619 04:50:40.872426 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:40.872435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:40.872846 (+ 411us) server_negotiation.cc:407] Connection header received
0619 04:50:40.873064 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:40.873069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:40.873157 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:40.873255 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:40.874851 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:40.876046 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:40.877366 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:40.878050 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:40.878215 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:40.881323 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:40.881339 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:40.881349 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:40.881385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:40.886143 (+ 4758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:40.887414 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:40.887422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:40.887427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:40.887678 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:40.888421 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:40.888429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:40.888435 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:40.888790 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:40.889160 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:40.889672 (+ 512us) server_negotiation.cc:299] Negotiation successful
0619 04:50:40.889934 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260619 04:50:40.892505 25187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:40.900923 25187 catalog_manager.cc:1347] Generated new cluster ID: 80c98eb5e7fd404096f376e873ee8d9f
I20260619 04:50:40.901038 25187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:40.920779 25187 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:40.922622 25187 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:40.934204 25187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ade07e2713fc49f2acec06e481b27603: Generated new TSK 0
I20260619 04:50:40.935101 25187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:40.944696 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25104
I20260619 04:50:40.945853 25166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:41.036422 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 25104
W20260619 04:50:41.053859 25089 connection.cc:535] client connection to 127.0.0.1:46157 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46157): Connection reset by peer (error 104)
2026-06-19T04:50:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:41 dist-test-slave-2qp8 krb5kdc[25203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:41 dist-test-slave-2qp8 krb5kdc[25203](info): set up 2 sockets
Jun 19 04:50:41 dist-test-slave-2qp8 krb5kdc[25203](info): commencing operation
krb5kdc: starting...
W20260619 04:50:42.205626 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:42 dist-test-slave-2qp8 krb5kdc[25203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:42.461453 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46541
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:42.782357 25216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:42.782635 25216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:42.782722 25216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:42.789800 25216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:42.789901 25216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:42.789947 25216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:42.790010 25216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:42.790053 25216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:42.799556 25216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:42.800853 25216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:42.802114 25216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:42.813409 25221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:42.813383 25222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:42.813439 25224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:42.814348 25216 server_base.cc:1034] running on GCE node
I20260619 04:50:42.815116 25216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:42.816520 25216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:42.817757 25216 hybrid_clock.cc:648] HybridClock initialized: now 1781844642817712 us; error 33 us; skew 500 ppm
I20260619 04:50:42.818127 25216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:42 dist-test-slave-2qp8 krb5kdc[25203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:42.821213 25216 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:42.822631 25216 webserver.cc:460] Webserver started at http://127.0.0.1:43387/ using document root <none> and password file <none>
I20260619 04:50:42.823297 25216 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:42.823426 25216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:42.823701 25216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:42.826699 25216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "f4da049ac0ab40f9b4ecf54c478634f5"
format_stamp: "Formatted at 2026-06-19 04:50:42 on dist-test-slave-2qp8"
server_key: "adfed2ae7831a0a9314edfcc4b62de83"
server_key_iv: "63d2c052fad58cc8d0466f2d3dc30d12"
server_key_version: "encryptionkey@0"
I20260619 04:50:42.827447 25216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "f4da049ac0ab40f9b4ecf54c478634f5"
format_stamp: "Formatted at 2026-06-19 04:50:42 on dist-test-slave-2qp8"
server_key: "adfed2ae7831a0a9314edfcc4b62de83"
server_key_iv: "63d2c052fad58cc8d0466f2d3dc30d12"
server_key_version: "encryptionkey@0"
I20260619 04:50:42.832593 25216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:50:42.835922 25231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:42.837172 25216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:42.837363 25216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "f4da049ac0ab40f9b4ecf54c478634f5"
format_stamp: "Formatted at 2026-06-19 04:50:42 on dist-test-slave-2qp8"
server_key: "adfed2ae7831a0a9314edfcc4b62de83"
server_key_iv: "63d2c052fad58cc8d0466f2d3dc30d12"
server_key_version: "encryptionkey@0"
I20260619 04:50:42.837579 25216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:42.886178 25216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:42.890669 25216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:42.890985 25216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:42.908887 25216 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46541
I20260619 04:50:42.908917 25282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46541 every 8 connection(s)
I20260619 04:50:42.910272 25216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:42.913998 25283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:42.914211 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 25216
I20260619 04:50:42.914386 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:42.914841 4046 external_mini_cluster.cc:1428] Setting key 87d4f884521b8a831b64f5e66148f4a9
I20260619 04:50:42.927443 25283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5: Bootstrap starting.
I20260619 04:50:42.927811 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:42.916547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:34473 (local address 127.0.0.1:46541)
0619 04:50:42.917341 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:42.917352 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:42.917394 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:50:42.918892 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:42.918917 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:42.919308 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:42.919621 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:42.920214 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:42.921556 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:42.923306 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:42.924012 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:42.924212 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:42.925476 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:42.925483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:42.925496 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:42.926069 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:42.926142 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:42.926169 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:42.926287 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:42.926612 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 04:50:42.926810 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":366,"threads_started":1}
I20260619 04:50:42.931085 25283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:42.931640 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:42.932252 25283 log.cc:824] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:42.935055 25283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5: No bootstrap required, opened a new log
I20260619 04:50:42.942534 25286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:42.936398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:59721 (local address 127.0.0.1:46541)
0619 04:50:42.936539 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:42.936544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:42.936959 (+ 415us) server_negotiation.cc:407] Connection header received
0619 04:50:42.937156 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:42.937160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:42.937218 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:42.937329 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:42.937812 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:42.938577 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:42.939915 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:42.940462 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:42.940649 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:42.941826 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:42.941831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:42.941834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:42.941850 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:42.941889 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:42.941906 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:42.942029 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:42.942214 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 04:50:42.942266 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 04:50:42.943740 25283 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4da049ac0ab40f9b4ecf54c478634f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46541 } }
I20260619 04:50:42.944317 25283 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:42.944420 25283 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4da049ac0ab40f9b4ecf54c478634f5, State: Initialized, Role: FOLLOWER
I20260619 04:50:42.945086 25283 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4da049ac0ab40f9b4ecf54c478634f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46541 } }
I20260619 04:50:42.945327 25283 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:42.945461 25283 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:42.945628 25283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:42.947999 25283 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4da049ac0ab40f9b4ecf54c478634f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46541 } }
I20260619 04:50:42.948529 25283 leader_election.cc:302] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4da049ac0ab40f9b4ecf54c478634f5; no voters:
I20260619 04:50:42.949071 25283 leader_election.cc:288] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:42.949247 25293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:42.949914 25293 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [term 1 LEADER]: Becoming Leader. State: Replica: f4da049ac0ab40f9b4ecf54c478634f5, State: Running, Role: LEADER
I20260619 04:50:42.950423 25293 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4da049ac0ab40f9b4ecf54c478634f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46541 } }
I20260619 04:50:42.951087 25283 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:42.953311 25295 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4da049ac0ab40f9b4ecf54c478634f5. Latest consensus state: current_term: 1 leader_uuid: "f4da049ac0ab40f9b4ecf54c478634f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4da049ac0ab40f9b4ecf54c478634f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46541 } } }
I20260619 04:50:42.953207 25294 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4da049ac0ab40f9b4ecf54c478634f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4da049ac0ab40f9b4ecf54c478634f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46541 } } }
I20260619 04:50:42.953715 25295 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:42.953740 25294 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:42.956182 25301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:42.960192 25301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:42.968719 25301 catalog_manager.cc:1347] Generated new cluster ID: a7edc6818aa24b05aa25eed2a8078557
I20260619 04:50:42.968825 25301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:42.995983 25301 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:42.997313 25301 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:43.003504 25301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4da049ac0ab40f9b4ecf54c478634f5: Generated new TSK 0
I20260619 04:50:43.004666 25301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:43.013324 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25216
W20260619 04:50:43.014707 25278 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.91:34473
W20260619 04:50:43.015416 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25216: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:43.015568 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 25216
W20260619 04:50:43.029814 25201 connection.cc:535] client connection to 127.0.0.1:46541 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46541): Connection reset by peer (error 104)
2026-06-19T04:50:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1977 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:43 dist-test-slave-2qp8 krb5kdc[25314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:43 dist-test-slave-2qp8 krb5kdc[25314](info): set up 2 sockets
Jun 19 04:50:43 dist-test-slave-2qp8 krb5kdc[25314](info): commencing operation
krb5kdc: starting...
W20260619 04:50:44.120747 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:44 dist-test-slave-2qp8 krb5kdc[25314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:44.377050 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:44.702482 25327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:44.702782 25327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:44.702943 25327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:44.710116 25327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:44.710232 25327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:44.710341 25327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:44.710448 25327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:44.710536 25327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:44.720042 25327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:44.721339 25327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:44.722643 25327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:44.734179 25332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:44.734141 25333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:44.734230 25335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:44.734763 25327 server_base.cc:1034] running on GCE node
I20260619 04:50:44.735496 25327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:44.736862 25327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:44.738086 25327 hybrid_clock.cc:648] HybridClock initialized: now 1781844644738038 us; error 37 us; skew 500 ppm
I20260619 04:50:44.738405 25327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:44 dist-test-slave-2qp8 krb5kdc[25314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:44.745123 25327 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:44.746515 25327 webserver.cc:460] Webserver started at http://127.0.0.1:41619/ using document root <none> and password file <none>
I20260619 04:50:44.747126 25327 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:44.747223 25327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:44.747493 25327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:44.750607 25327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "2da47287fdbf4833b045bfbe1afb2021"
format_stamp: "Formatted at 2026-06-19 04:50:44 on dist-test-slave-2qp8"
server_key: "ac7bf88829014f55bcffdeba4b592ac4"
server_key_iv: "be4af17a8420a6fdb50bebf5d305f899"
server_key_version: "encryptionkey@0"
I20260619 04:50:44.751318 25327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "2da47287fdbf4833b045bfbe1afb2021"
format_stamp: "Formatted at 2026-06-19 04:50:44 on dist-test-slave-2qp8"
server_key: "ac7bf88829014f55bcffdeba4b592ac4"
server_key_iv: "be4af17a8420a6fdb50bebf5d305f899"
server_key_version: "encryptionkey@0"
I20260619 04:50:44.756255 25327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 04:50:44.759436 25342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:44.760792 25327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:44.760928 25327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "2da47287fdbf4833b045bfbe1afb2021"
format_stamp: "Formatted at 2026-06-19 04:50:44 on dist-test-slave-2qp8"
server_key: "ac7bf88829014f55bcffdeba4b592ac4"
server_key_iv: "be4af17a8420a6fdb50bebf5d305f899"
server_key_version: "encryptionkey@0"
I20260619 04:50:44.761082 25327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:44.766310 25327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:44.770431 25327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:44.770669 25327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:44.787160 25327 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34117
I20260619 04:50:44.787209 25393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34117 every 8 connection(s)
I20260619 04:50:44.788481 25327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:44.790210 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 25327
I20260619 04:50:44.790407 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:44.790853 4046 external_mini_cluster.cc:1428] Setting key 8651d2a2032b657f96d5f490617300ee
I20260619 04:50:44.792484 25394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:44.799237 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:44.792592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:33003 (local address 127.0.0.1:34117)
0619 04:50:44.793412 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:44.793443 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:44.793497 (+ 54us) server_negotiation.cc:407] Connection header received
0619 04:50:44.795470 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:44.795495 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:44.795955 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:44.796407 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:44.797354 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:44.797364 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:44.797381 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:44.797403 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:44.797485 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:44.797519 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:44.797713 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:44.797858 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 04:50:44.798050 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":311,"threads_started":1}
I20260619 04:50:44.802798 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:44.805838 25394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021: Bootstrap starting.
I20260619 04:50:44.809247 25394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:44.809504 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:44.807083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:45265 (local address 127.0.0.1:34117)
0619 04:50:44.807302 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:44.807310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:44.807608 (+ 298us) server_negotiation.cc:407] Connection header received
0619 04:50:44.807796 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:44.807800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:44.807843 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:44.807944 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:44.808887 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:44.808893 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:44.808897 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:44.808913 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:44.808953 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:44.808967 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:44.809074 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:44.809192 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 04:50:44.809260 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 04:50:44.810592 25394 log.cc:824] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:44.813313 25394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021: No bootstrap required, opened a new log
I20260619 04:50:44.820545 25394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da47287fdbf4833b045bfbe1afb2021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34117 } }
I20260619 04:50:44.820865 25394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:44.820935 25394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da47287fdbf4833b045bfbe1afb2021, State: Initialized, Role: FOLLOWER
I20260619 04:50:44.821455 25394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da47287fdbf4833b045bfbe1afb2021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34117 } }
I20260619 04:50:44.821642 25394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:44.821748 25394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:44.821879 25394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:44.823771 25394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da47287fdbf4833b045bfbe1afb2021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34117 } }
I20260619 04:50:44.824273 25394 leader_election.cc:302] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2da47287fdbf4833b045bfbe1afb2021; no voters:
I20260619 04:50:44.824940 25394 leader_election.cc:288] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:44.825028 25404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:44.825724 25404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [term 1 LEADER]: Becoming Leader. State: Replica: 2da47287fdbf4833b045bfbe1afb2021, State: Running, Role: LEADER
I20260619 04:50:44.826303 25404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da47287fdbf4833b045bfbe1afb2021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34117 } }
I20260619 04:50:44.826936 25394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:44.829249 25405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2da47287fdbf4833b045bfbe1afb2021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da47287fdbf4833b045bfbe1afb2021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34117 } } }
I20260619 04:50:44.829418 25406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2da47287fdbf4833b045bfbe1afb2021. Latest consensus state: current_term: 1 leader_uuid: "2da47287fdbf4833b045bfbe1afb2021" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da47287fdbf4833b045bfbe1afb2021" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34117 } } }
I20260619 04:50:44.829744 25405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:44.829804 25406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:44.831204 25413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:44.836750 25413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:44.845832 25413 catalog_manager.cc:1347] Generated new cluster ID: 8f8176b4718a469682eaf4fd84fac767
I20260619 04:50:44.845945 25413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:44.876803 25413 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:44.877226 25413 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:44.893438 25413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2da47287fdbf4833b045bfbe1afb2021: Generated new TSK 0
I20260619 04:50:44.894224 25413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:44.906143 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25327
W20260619 04:50:44.907187 25389 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.91:33003
W20260619 04:50:44.907874 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25327: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:44.908013 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 25327
2026-06-19T04:50:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1890 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:44 dist-test-slave-2qp8 krb5kdc[25425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:45 dist-test-slave-2qp8 krb5kdc[25425](info): set up 2 sockets
Jun 19 04:50:45 dist-test-slave-2qp8 krb5kdc[25425](info): commencing operation
krb5kdc: starting...
W20260619 04:50:45.984536 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:46 dist-test-slave-2qp8 krb5kdc[25425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:46.243103 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:46.571501 25438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:46.571827 25438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:46.571983 25438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:46.578938 25438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:46.579088 25438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:46.579195 25438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:46.579299 25438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:46.579387 25438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:46.588958 25438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:46.590265 25438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:46.591583 25438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:46.603255 25444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:46.603257 25443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:46.603309 25446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:46.604199 25438 server_base.cc:1034] running on GCE node
I20260619 04:50:46.605041 25438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:46.606449 25438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:46.607668 25438 hybrid_clock.cc:648] HybridClock initialized: now 1781844646607634 us; error 35 us; skew 500 ppm
I20260619 04:50:46.608019 25438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:46 dist-test-slave-2qp8 krb5kdc[25425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:46.611007 25438 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:46.612439 25438 webserver.cc:460] Webserver started at http://127.0.0.1:40971/ using document root <none> and password file <none>
I20260619 04:50:46.613122 25438 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:46.613257 25438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:46.613548 25438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:46.616564 25438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "7cf412605e7a4b97987fb74c852c8a84"
format_stamp: "Formatted at 2026-06-19 04:50:46 on dist-test-slave-2qp8"
server_key: "9f8bb82c3b5bcddeb5798b6b941dfe28"
server_key_iv: "81bebc0734f27bb485f8dc1a2316a9bd"
server_key_version: "encryptionkey@0"
I20260619 04:50:46.617317 25438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "7cf412605e7a4b97987fb74c852c8a84"
format_stamp: "Formatted at 2026-06-19 04:50:46 on dist-test-slave-2qp8"
server_key: "9f8bb82c3b5bcddeb5798b6b941dfe28"
server_key_iv: "81bebc0734f27bb485f8dc1a2316a9bd"
server_key_version: "encryptionkey@0"
I20260619 04:50:46.622264 25438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:50:46.625620 25453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:46.626896 25438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260619 04:50:46.627069 25438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "7cf412605e7a4b97987fb74c852c8a84"
format_stamp: "Formatted at 2026-06-19 04:50:46 on dist-test-slave-2qp8"
server_key: "9f8bb82c3b5bcddeb5798b6b941dfe28"
server_key_iv: "81bebc0734f27bb485f8dc1a2316a9bd"
server_key_version: "encryptionkey@0"
I20260619 04:50:46.627257 25438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:46.654708 25438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:46.658937 25438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:46.659262 25438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:46.678542 25438 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36805
I20260619 04:50:46.678571 25504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36805 every 8 connection(s)
I20260619 04:50:46.680004 25438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:46.683391 25505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:46.687026 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 25438
I20260619 04:50:46.687211 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:46.687748 4046 external_mini_cluster.cc:1428] Setting key b5a192061171e7f49f53a141be37d402
I20260619 04:50:46.695720 25505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84: Bootstrap starting.
I20260619 04:50:46.699309 25505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:46.700654 25505 log.cc:824] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84: Log is configured to *not* fsync() on all Append() calls
Jun 19 04:50:46 dist-test-slave-2qp8 krb5kdc[25425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 04:50:46.703487 25505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84: No bootstrap required, opened a new log
I20260619 04:50:46.710301 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:46.689611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:46261 (local address 127.0.0.1:36805)
0619 04:50:46.690317 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:46.690329 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:46.690367 (+ 38us) server_negotiation.cc:407] Connection header received
0619 04:50:46.691895 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:46.691915 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:46.692386 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:46.692686 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:46.694340 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:46.695604 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:46.697333 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:46.697969 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:46.698092 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:46.703076 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:46.703084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:46.703098 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:46.703132 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:46.706274 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:46.707033 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:46.707040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:46.707045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:46.707197 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:46.707645 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:46.707649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:46.707651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:46.708215 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:46.708548 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:46.709033 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 04:50:46.709337 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":325,"threads_started":1}
I20260619 04:50:46.711232 25505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf412605e7a4b97987fb74c852c8a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260619 04:50:46.711586 25505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:46.711683 25505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf412605e7a4b97987fb74c852c8a84, State: Initialized, Role: FOLLOWER
I20260619 04:50:46.712327 25505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf412605e7a4b97987fb74c852c8a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260619 04:50:46.712589 25505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:46.712750 25505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:46.712911 25505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:46.714138 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:46.715328 25505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf412605e7a4b97987fb74c852c8a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260619 04:50:46.715826 25505 leader_election.cc:302] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cf412605e7a4b97987fb74c852c8a84; no voters:
I20260619 04:50:46.716475 25505 leader_election.cc:288] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:46.716626 25510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:46.717345 25510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [term 1 LEADER]: Becoming Leader. State: Replica: 7cf412605e7a4b97987fb74c852c8a84, State: Running, Role: LEADER
I20260619 04:50:46.718080 25510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf412605e7a4b97987fb74c852c8a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } }
I20260619 04:50:46.719782 25505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:46.722078 25514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cf412605e7a4b97987fb74c852c8a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf412605e7a4b97987fb74c852c8a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } } }
I20260619 04:50:46.722478 25514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:46.722047 25513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cf412605e7a4b97987fb74c852c8a84. Latest consensus state: current_term: 1 leader_uuid: "7cf412605e7a4b97987fb74c852c8a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cf412605e7a4b97987fb74c852c8a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36805 } } }
I20260619 04:50:46.723173 25513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:46.723312 25520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:46.732079 25520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:46.751741 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:46.720320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:32937 (local address 127.0.0.1:36805)
0619 04:50:46.720545 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:46.720553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:46.728499 (+ 7946us) server_negotiation.cc:407] Connection header received
0619 04:50:46.729057 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:46.729065 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:46.729178 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:46.729266 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:46.731749 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:46.733411 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:46.736021 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:46.736950 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:46.737079 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:46.741424 (+ 4345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:46.741433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:46.741440 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:46.741490 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:46.746306 (+ 4816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:46.747546 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:46.747554 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:46.747560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:46.747740 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:46.748465 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:46.748473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:46.748479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:46.749051 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:46.749381 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:46.750942 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0619 04:50:46.751187 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 04:50:46.753746 25520 catalog_manager.cc:1347] Generated new cluster ID: d34dc9f36ed546bdb75fd5ce071f7ee5
I20260619 04:50:46.753858 25520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:46.775063 25520 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:46.776917 25520 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:46.789644 25520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cf412605e7a4b97987fb74c852c8a84: Generated new TSK 0
I20260619 04:50:46.790467 25520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:46.798945 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25438
I20260619 04:50:46.800441 25500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:46.890064 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 25438
W20260619 04:50:46.907295 25423 connection.cc:535] client connection to 127.0.0.1:36805 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36805): Connection reset by peer (error 104)
2026-06-19T04:50:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:46 dist-test-slave-2qp8 krb5kdc[25537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:46 dist-test-slave-2qp8 krb5kdc[25537](info): set up 2 sockets
Jun 19 04:50:46 dist-test-slave-2qp8 krb5kdc[25537](info): commencing operation
krb5kdc: starting...
W20260619 04:50:48.044693 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:48 dist-test-slave-2qp8 krb5kdc[25537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:48.299463 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:48.624914 25550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:48.625254 25550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:48.625413 25550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:48.632519 25550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:48.632656 25550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:48.632750 25550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:48.632862 25550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:48.632946 25550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:48.642215 25550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:48.643491 25550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:48.644816 25550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:48.656571 25556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:48.656637 25558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:48.656612 25555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:48.657133 25550 server_base.cc:1034] running on GCE node
I20260619 04:50:48.657972 25550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:48.659350 25550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:48.660570 25550 hybrid_clock.cc:648] HybridClock initialized: now 1781844648660538 us; error 46 us; skew 500 ppm
I20260619 04:50:48.660907 25550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:48 dist-test-slave-2qp8 krb5kdc[25537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:48.663996 25550 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:48.665457 25550 webserver.cc:460] Webserver started at http://127.0.0.1:33597/ using document root <none> and password file <none>
I20260619 04:50:48.666119 25550 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:48.666265 25550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:48.666574 25550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:48.669590 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "a83208aa13064b29841e7123231b4c3f"
format_stamp: "Formatted at 2026-06-19 04:50:48 on dist-test-slave-2qp8"
server_key: "5dd54ae315094c82d0bd2368b3b068ab"
server_key_iv: "6c7e02db877bf8dce5b105d092d6e2c7"
server_key_version: "encryptionkey@0"
I20260619 04:50:48.670360 25550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "a83208aa13064b29841e7123231b4c3f"
format_stamp: "Formatted at 2026-06-19 04:50:48 on dist-test-slave-2qp8"
server_key: "5dd54ae315094c82d0bd2368b3b068ab"
server_key_iv: "6c7e02db877bf8dce5b105d092d6e2c7"
server_key_version: "encryptionkey@0"
I20260619 04:50:48.675746 25550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 04:50:48.679064 25565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:48.680495 25550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:48.680667 25550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "a83208aa13064b29841e7123231b4c3f"
format_stamp: "Formatted at 2026-06-19 04:50:48 on dist-test-slave-2qp8"
server_key: "5dd54ae315094c82d0bd2368b3b068ab"
server_key_iv: "6c7e02db877bf8dce5b105d092d6e2c7"
server_key_version: "encryptionkey@0"
I20260619 04:50:48.680876 25550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:48.701102 25550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:48.705094 25550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:48.705377 25550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:48.723399 25550 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44031
I20260619 04:50:48.723433 25616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44031 every 8 connection(s)
I20260619 04:50:48.724789 25550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:48.728505 25617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:48.733084 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 25550
I20260619 04:50:48.733294 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:48.733711 4046 external_mini_cluster.cc:1428] Setting key 77ff60c93f2366a8fa970942999a4281
I20260619 04:50:48.739212 25617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f: Bootstrap starting.
I20260619 04:50:48.742767 25617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:48.743993 25617 log.cc:824] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:48.747236 25617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f: No bootstrap required, opened a new log
I20260619 04:50:48.747257 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:48.735592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:46763 (local address 127.0.0.1:44031)
0619 04:50:48.736416 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:48.736426 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:48.736472 (+ 46us) server_negotiation.cc:407] Connection header received
0619 04:50:48.738117 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:48.738139 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:48.738534 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:48.738851 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:48.739464 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:48.740836 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:48.742680 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:48.743432 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:48.743558 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:48.744960 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:48.744974 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:48.744999 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:48.745733 (+ 734us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:48.745807 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:48.745831 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:48.745943 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:48.746218 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 04:50:48.746374 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":369,"threads_started":1}
I20260619 04:50:48.751070 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:48.755263 25617 raft_consensus.cc:348] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83208aa13064b29841e7123231b4c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44031 } }
I20260619 04:50:48.755784 25617 raft_consensus.cc:374] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:48.755940 25617 raft_consensus.cc:729] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83208aa13064b29841e7123231b4c3f, State: Initialized, Role: FOLLOWER
I20260619 04:50:48.756626 25617 consensus_queue.cc:260] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83208aa13064b29841e7123231b4c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44031 } }
I20260619 04:50:48.756877 25617 raft_consensus.cc:388] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:48.757057 25617 raft_consensus.cc:482] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:48.757256 25617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:48.760026 25617 raft_consensus.cc:504] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83208aa13064b29841e7123231b4c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44031 } }
I20260619 04:50:48.760618 25617 leader_election.cc:302] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a83208aa13064b29841e7123231b4c3f; no voters:
I20260619 04:50:48.761432 25617 leader_election.cc:288] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:48.761606 25627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:48.762866 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:48.755428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:60685 (local address 127.0.0.1:44031)
0619 04:50:48.755609 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:48.755619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:48.756052 (+ 433us) server_negotiation.cc:407] Connection header received
0619 04:50:48.756311 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:48.756315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:48.756378 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:48.756479 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:48.756970 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:48.758145 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:48.759586 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:48.760290 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:48.760453 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:48.761614 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:48.761620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:48.761624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:48.761643 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:48.761721 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:48.761747 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:48.761906 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:48.762496 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 04:50:48.762548 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 04:50:48.763304 25627 raft_consensus.cc:686] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [term 1 LEADER]: Becoming Leader. State: Replica: a83208aa13064b29841e7123231b4c3f, State: Running, Role: LEADER
I20260619 04:50:48.763892 25617 sys_catalog.cc:564] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:48.763893 25627 consensus_queue.cc:237] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83208aa13064b29841e7123231b4c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44031 } }
I20260619 04:50:48.766872 25629 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a83208aa13064b29841e7123231b4c3f. Latest consensus state: current_term: 1 leader_uuid: "a83208aa13064b29841e7123231b4c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83208aa13064b29841e7123231b4c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44031 } } }
I20260619 04:50:48.767266 25629 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:48.767159 25628 sys_catalog.cc:455] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a83208aa13064b29841e7123231b4c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a83208aa13064b29841e7123231b4c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44031 } } }
I20260619 04:50:48.767397 25628 sys_catalog.cc:458] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:48.774073 25635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:48.779076 25635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:48.788218 25635 catalog_manager.cc:1347] Generated new cluster ID: d117b7bd89ac499c975619d54098229f
I20260619 04:50:48.788367 25635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:48.796722 25635 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:48.798506 25635 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:48.807191 25635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a83208aa13064b29841e7123231b4c3f: Generated new TSK 0
I20260619 04:50:48.808195 25635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:48.823599 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25550
W20260619 04:50:48.824793 25612 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.91:46763
W20260619 04:50:48.825549 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25550: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:48.825686 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 25550
W20260619 04:50:48.842367 25535 connection.cc:535] client connection to 127.0.0.1:44031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44031): Connection reset by peer (error 104)
2026-06-19T04:50:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:48 dist-test-slave-2qp8 krb5kdc[25648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:48 dist-test-slave-2qp8 krb5kdc[25648](info): set up 2 sockets
Jun 19 04:50:48 dist-test-slave-2qp8 krb5kdc[25648](info): commencing operation
krb5kdc: starting...
W20260619 04:50:49.920725 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:50 dist-test-slave-2qp8 krb5kdc[25648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 04:50:50.183813 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42157
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:50.507611 25661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:50.507895 25661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:50.508054 25661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:50.515285 25661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:50.515422 25661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:50.515522 25661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:50.515631 25661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:50.515693 25661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:50.525672 25661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42157
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:50.526932 25661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:50.528232 25661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:50.539805 25666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:50.539914 25667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:50.539901 25669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:50.540908 25661 server_base.cc:1034] running on GCE node
I20260619 04:50:50.541653 25661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:50.543054 25661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:50.544278 25661 hybrid_clock.cc:648] HybridClock initialized: now 1781844650544233 us; error 41 us; skew 500 ppm
I20260619 04:50:50.544639 25661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:50 dist-test-slave-2qp8 krb5kdc[25648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844650, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:50.547715 25661 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 04:50:50.549129 25661 webserver.cc:460] Webserver started at http://127.0.0.1:38057/ using document root <none> and password file <none>
I20260619 04:50:50.549737 25661 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:50.549831 25661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:50.550071 25661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:50.552929 25661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "5cd8fa1fc31746f5849ea16ac47d8f08"
format_stamp: "Formatted at 2026-06-19 04:50:50 on dist-test-slave-2qp8"
server_key: "c0ebfecbfe0b32a7f1fd9d1144d2bc86"
server_key_iv: "43f2bfee780ebe3933df613c9ab9d58a"
server_key_version: "encryptionkey@0"
I20260619 04:50:50.553629 25661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "5cd8fa1fc31746f5849ea16ac47d8f08"
format_stamp: "Formatted at 2026-06-19 04:50:50 on dist-test-slave-2qp8"
server_key: "c0ebfecbfe0b32a7f1fd9d1144d2bc86"
server_key_iv: "43f2bfee780ebe3933df613c9ab9d58a"
server_key_version: "encryptionkey@0"
I20260619 04:50:50.558420 25661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 04:50:50.561797 25676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:50.563187 25661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:50:50.563359 25661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "5cd8fa1fc31746f5849ea16ac47d8f08"
format_stamp: "Formatted at 2026-06-19 04:50:50 on dist-test-slave-2qp8"
server_key: "c0ebfecbfe0b32a7f1fd9d1144d2bc86"
server_key_iv: "43f2bfee780ebe3933df613c9ab9d58a"
server_key_version: "encryptionkey@0"
I20260619 04:50:50.563575 25661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:50.569092 25661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:50.573350 25661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:50.573637 25661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:50.591567 25661 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42157
I20260619 04:50:50.591604 25727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42157 every 8 connection(s)
I20260619 04:50:50.592993 25661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:50.596611 25728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:50.599807 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 25661
I20260619 04:50:50.599982 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:50.600566 4046 external_mini_cluster.cc:1428] Setting key eac1d4e1d421188ddbd7b73b6ef896ac
I20260619 04:50:50.607879 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:50.602332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:50629 (local address 127.0.0.1:42157)
0619 04:50:50.603061 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:50.603072 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:50.603109 (+ 37us) server_negotiation.cc:407] Connection header received
0619 04:50:50.604754 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:50.604774 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:50.605110 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:50.605431 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:50.606436 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:50.606442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:50.606457 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:50.606476 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:50.606530 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:50.606551 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:50.606724 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:50.606821 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 04:50:50.607010 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":336,"threads_started":1}
I20260619 04:50:50.608578 25728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08: Bootstrap starting.
I20260619 04:50:50.611373 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:50.611616 25728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:50.612623 25728 log.cc:824] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:50.615537 25728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08: No bootstrap required, opened a new log
I20260619 04:50:50.618618 25731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:50.616262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:39519 (local address 127.0.0.1:42157)
0619 04:50:50.616434 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:50.616440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:50.616751 (+ 311us) server_negotiation.cc:407] Connection header received
0619 04:50:50.616942 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:50.616946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:50.616988 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:50.617084 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:50.617984 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:50.617989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:50.617992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:50.618005 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:50.618041 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:50.618057 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:50.618177 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:50.618285 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 04:50:50.618360 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 04:50:50.623600 25728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd8fa1fc31746f5849ea16ac47d8f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42157 } }
I20260619 04:50:50.623991 25728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:50.624109 25728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cd8fa1fc31746f5849ea16ac47d8f08, State: Initialized, Role: FOLLOWER
I20260619 04:50:50.624672 25728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd8fa1fc31746f5849ea16ac47d8f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42157 } }
I20260619 04:50:50.624922 25728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:50.625061 25728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:50.625247 25728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:50.627341 25728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd8fa1fc31746f5849ea16ac47d8f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42157 } }
I20260619 04:50:50.627849 25728 leader_election.cc:302] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cd8fa1fc31746f5849ea16ac47d8f08; no voters:
I20260619 04:50:50.628487 25728 leader_election.cc:288] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:50.628613 25738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:50.629266 25738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [term 1 LEADER]: Becoming Leader. State: Replica: 5cd8fa1fc31746f5849ea16ac47d8f08, State: Running, Role: LEADER
I20260619 04:50:50.629791 25738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd8fa1fc31746f5849ea16ac47d8f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42157 } }
I20260619 04:50:50.630529 25728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:50.632509 25740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cd8fa1fc31746f5849ea16ac47d8f08. Latest consensus state: current_term: 1 leader_uuid: "5cd8fa1fc31746f5849ea16ac47d8f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd8fa1fc31746f5849ea16ac47d8f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42157 } } }
I20260619 04:50:50.633028 25740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:50.632495 25739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cd8fa1fc31746f5849ea16ac47d8f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cd8fa1fc31746f5849ea16ac47d8f08" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42157 } } }
I20260619 04:50:50.639225 25739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:50.640961 25746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:50.646191 25746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:50.654896 25746 catalog_manager.cc:1347] Generated new cluster ID: 4a9be805a4e14ea9bc7adbfd7a810110
I20260619 04:50:50.655015 25746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:50.674656 25746 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:50.675087 25746 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:50.688719 25746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cd8fa1fc31746f5849ea16ac47d8f08: Generated new TSK 0
I20260619 04:50:50.689503 25746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:50.700215 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25661
W20260619 04:50:50.701292 25723 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.91:50629
W20260619 04:50:50.702013 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25661: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:50.702170 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 25661
2026-06-19T04:50:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1871 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:50 dist-test-slave-2qp8 krb5kdc[25759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:50 dist-test-slave-2qp8 krb5kdc[25759](info): set up 2 sockets
Jun 19 04:50:50 dist-test-slave-2qp8 krb5kdc[25759](info): commencing operation
krb5kdc: starting...
W20260619 04:50:51.796716 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:51 dist-test-slave-2qp8 krb5kdc[25759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:50:52.043430 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44101
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:52.365777 25772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:52.366047 25772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:52.366218 25772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:52.373414 25772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:52.373520 25772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:52.373565 25772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:52.373651 25772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:52.373728 25772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:52.383412 25772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:52.384701 25772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:52.385996 25772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:52.397364 25777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:52.397404 25780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:52.397441 25778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:52.398439 25772 server_base.cc:1034] running on GCE node
I20260619 04:50:52.399147 25772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:52.400454 25772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:52.401649 25772 hybrid_clock.cc:648] HybridClock initialized: now 1781844652401602 us; error 32 us; skew 500 ppm
I20260619 04:50:52.401998 25772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:52 dist-test-slave-2qp8 krb5kdc[25759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844652, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:52.405161 25772 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:50:52.406605 25772 webserver.cc:460] Webserver started at http://127.3.243.190:41133/ using document root <none> and password file <none>
I20260619 04:50:52.407270 25772 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:52.407394 25772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:52.407698 25772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:52.410667 25772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "229e60ac7e6048b98ad630e90f8262f7"
format_stamp: "Formatted at 2026-06-19 04:50:52 on dist-test-slave-2qp8"
server_key: "147f9403193da969cc7d802853b04084"
server_key_iv: "89c17f043e004df5b9243f635e193d6c"
server_key_version: "encryptionkey@0"
I20260619 04:50:52.411492 25772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "229e60ac7e6048b98ad630e90f8262f7"
format_stamp: "Formatted at 2026-06-19 04:50:52 on dist-test-slave-2qp8"
server_key: "147f9403193da969cc7d802853b04084"
server_key_iv: "89c17f043e004df5b9243f635e193d6c"
server_key_version: "encryptionkey@0"
I20260619 04:50:52.416798 25772 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 04:50:52.420183 25787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:52.421460 25772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:50:52.421626 25772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "229e60ac7e6048b98ad630e90f8262f7"
format_stamp: "Formatted at 2026-06-19 04:50:52 on dist-test-slave-2qp8"
server_key: "147f9403193da969cc7d802853b04084"
server_key_iv: "89c17f043e004df5b9243f635e193d6c"
server_key_version: "encryptionkey@0"
I20260619 04:50:52.421802 25772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:52.437786 25772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:52.442114 25772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:52.442390 25772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:52.460521 25772 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44101
I20260619 04:50:52.460528 25838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44101 every 8 connection(s)
I20260619 04:50:52.461802 25772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:52.465327 25839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:52.466912 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 25772
I20260619 04:50:52.467144 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:52.467622 4046 external_mini_cluster.cc:1428] Setting key 3e55be2933178343e657aa02799a6aae
I20260619 04:50:52.477875 25839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7: Bootstrap starting.
Jun 19 04:50:52 dist-test-slave-2qp8 krb5kdc[25759](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:52.481025 25839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:52.482287 25839 log.cc:824] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:52.485278 25839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7: No bootstrap required, opened a new log
I20260619 04:50:52.490037 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:52.469487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48109 (local address 127.3.243.190:44101)
0619 04:50:52.470261 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:52.470275 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:52.470316 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:50:52.471898 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:52.471922 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:52.472395 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:52.472786 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:52.474197 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:52.475497 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:52.476854 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:52.477530 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:52.477740 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:52.482351 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:52.482359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:52.482398 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:52.482466 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:52.485760 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:52.486762 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:52.486767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:52.486781 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:52.486938 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:52.487443 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:52.487447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:52.487449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:52.487962 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:52.488279 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:52.488700 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 04:50:52.489016 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":337,"threads_started":1}
I20260619 04:50:52.493801 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:52.493992 25839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229e60ac7e6048b98ad630e90f8262f7" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44101 } }
I20260619 04:50:52.494374 25839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:52.494470 25839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 229e60ac7e6048b98ad630e90f8262f7, State: Initialized, Role: FOLLOWER
I20260619 04:50:52.495090 25839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229e60ac7e6048b98ad630e90f8262f7" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44101 } }
I20260619 04:50:52.495358 25839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:52.495478 25839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:52.495630 25839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:52.498934 25839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229e60ac7e6048b98ad630e90f8262f7" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44101 } }
I20260619 04:50:52.499512 25839 leader_election.cc:302] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 229e60ac7e6048b98ad630e90f8262f7; no voters:
I20260619 04:50:52.500185 25839 leader_election.cc:288] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:52.500314 25849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:52.502132 25849 raft_consensus.cc:686] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [term 1 LEADER]: Becoming Leader. State: Replica: 229e60ac7e6048b98ad630e90f8262f7, State: Running, Role: LEADER
I20260619 04:50:52.502394 25839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:52.502668 25849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229e60ac7e6048b98ad630e90f8262f7" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44101 } }
I20260619 04:50:52.507644 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 229e60ac7e6048b98ad630e90f8262f7. Latest consensus state: current_term: 1 leader_uuid: "229e60ac7e6048b98ad630e90f8262f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229e60ac7e6048b98ad630e90f8262f7" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44101 } } }
I20260619 04:50:52.507982 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:52.506999 25850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "229e60ac7e6048b98ad630e90f8262f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "229e60ac7e6048b98ad630e90f8262f7" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44101 } } }
I20260619 04:50:52.508515 25850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:52.513394 25858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:52.518090 25842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:52.498955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43037 (local address 127.3.243.190:44101)
0619 04:50:52.499136 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:52.499142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:52.499635 (+ 493us) server_negotiation.cc:407] Connection header received
0619 04:50:52.499807 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:52.499810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:52.499866 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:52.499979 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:52.501847 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:52.503157 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:52.505315 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:52.506199 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:52.506362 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:52.509891 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:52.509901 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:52.509908 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:52.509958 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:52.514192 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:52.514959 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:52.514963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:52.514965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:52.515104 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:52.515576 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:52.515580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:52.515601 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:52.516054 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:52.517387 (+ 1333us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:52.517564 (+ 177us) server_negotiation.cc:299] Negotiation successful
0619 04:50:52.517746 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 04:50:52.519699 25858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:52.529860 25858 catalog_manager.cc:1347] Generated new cluster ID: dfe1d440a1db4ba6ab2fee891acb5c00
I20260619 04:50:52.529969 25858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:52.550518 25858 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:52.551821 25858 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:52.559027 25858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 229e60ac7e6048b98ad630e90f8262f7: Generated new TSK 0
I20260619 04:50:52.559787 25858 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:52.574376 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25772
I20260619 04:50:52.576043 25834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:52.667164 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 25772
W20260619 04:50:52.684397 25757 connection.cc:535] client connection to 127.3.243.190:44101 recv error: Network error: failed to read from TLS socket (remote: 127.3.243.190:44101): Connection reset by peer (error 104)
2026-06-19T04:50:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:52 dist-test-slave-2qp8 krb5kdc[25871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:52 dist-test-slave-2qp8 krb5kdc[25871](info): set up 2 sockets
Jun 19 04:50:52 dist-test-slave-2qp8 krb5kdc[25871](info): commencing operation
krb5kdc: starting...
W20260619 04:50:53.752885 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:53 dist-test-slave-2qp8 krb5kdc[25871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:50:53.997092 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:39499
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:41963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:39499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:54.324884 25884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:54.325233 25884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:54.325410 25884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:54.332337 25884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:54.332487 25884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:54.332566 25884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:54.332645 25884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:54.332707 25884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:54.342926 25884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:41963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:39499
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:39499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:54.344403 25884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:54.345795 25884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:54.358560 25890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:54.358536 25892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:54.358543 25889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:54.359452 25884 server_base.cc:1034] running on GCE node
I20260619 04:50:54.360330 25884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:54.361757 25884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:54.363087 25884 hybrid_clock.cc:648] HybridClock initialized: now 1781844654363046 us; error 63 us; skew 500 ppm
I20260619 04:50:54.363461 25884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:54 dist-test-slave-2qp8 krb5kdc[25871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844654, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:54.366729 25884 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:50:54.368057 25884 webserver.cc:460] Webserver started at http://127.3.243.190:39683/ using document root <none> and password file <none>
I20260619 04:50:54.368738 25884 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:54.368839 25884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:54.369104 25884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:54.372189 25884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "dc76968c19b047e0ad4dd77d745fc8d6"
format_stamp: "Formatted at 2026-06-19 04:50:54 on dist-test-slave-2qp8"
server_key: "8fd2aca029772016868840027aab6738"
server_key_iv: "8b9c7ddd3ea9e3da11a960c85d6ca2ff"
server_key_version: "encryptionkey@0"
I20260619 04:50:54.372903 25884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "dc76968c19b047e0ad4dd77d745fc8d6"
format_stamp: "Formatted at 2026-06-19 04:50:54 on dist-test-slave-2qp8"
server_key: "8fd2aca029772016868840027aab6738"
server_key_iv: "8b9c7ddd3ea9e3da11a960c85d6ca2ff"
server_key_version: "encryptionkey@0"
I20260619 04:50:54.377954 25884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:50:54.381490 25899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:54.382943 25884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:50:54.383141 25884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "dc76968c19b047e0ad4dd77d745fc8d6"
format_stamp: "Formatted at 2026-06-19 04:50:54 on dist-test-slave-2qp8"
server_key: "8fd2aca029772016868840027aab6738"
server_key_iv: "8b9c7ddd3ea9e3da11a960c85d6ca2ff"
server_key_version: "encryptionkey@0"
I20260619 04:50:54.383353 25884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:54.404428 25884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:54.408380 25884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:54.408610 25884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:54.426240 25884 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:39499
I20260619 04:50:54.426247 25950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:39499 every 8 connection(s)
I20260619 04:50:54.427645 25884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:54.430773 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 25884
I20260619 04:50:54.430972 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:54.431418 4046 external_mini_cluster.cc:1428] Setting key a5f8868a035d0a3caca26a2850814d12
I20260619 04:50:54.431668 25951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:54.447041 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:54.433227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43071 (local address 127.3.243.190:39499)
0619 04:50:54.434004 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:54.434020 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:54.434066 (+ 46us) server_negotiation.cc:407] Connection header received
0619 04:50:54.436703 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:54.436731 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:54.437298 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:54.437760 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:54.438302 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:54.440328 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:54.441761 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:54.442768 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:54.442949 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:54.444235 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:54.444245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:54.444263 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:54.445078 (+ 815us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:54.445170 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:54.445200 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:54.445360 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:54.445605 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 04:50:54.445773 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":317,"threads_started":1}
I20260619 04:50:54.447299 25951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6: Bootstrap starting.
I20260619 04:50:54.450520 25951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:54.450968 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:54.451712 25951 log.cc:824] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:54.455013 25951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6: No bootstrap required, opened a new log
I20260619 04:50:54.463002 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:54.455823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46121 (local address 127.3.243.190:39499)
0619 04:50:54.456000 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:54.456006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:54.456677 (+ 671us) server_negotiation.cc:407] Connection header received
0619 04:50:54.456811 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:54.456817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:54.456896 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:54.457005 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:54.457517 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:54.458694 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:54.460078 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:54.460827 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:54.460968 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:54.462155 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:54.462163 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:54.462173 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:54.462194 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:54.462242 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:54.462270 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:54.462400 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:54.462610 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 04:50:54.462660 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 04:50:54.464574 25951 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76968c19b047e0ad4dd77d745fc8d6" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39499 } }
I20260619 04:50:54.464934 25951 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:54.465052 25951 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc76968c19b047e0ad4dd77d745fc8d6, State: Initialized, Role: FOLLOWER
I20260619 04:50:54.465597 25951 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76968c19b047e0ad4dd77d745fc8d6" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39499 } }
I20260619 04:50:54.465813 25951 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:54.465963 25951 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:54.466126 25951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:54.468350 25951 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76968c19b047e0ad4dd77d745fc8d6" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39499 } }
I20260619 04:50:54.468842 25951 leader_election.cc:302] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc76968c19b047e0ad4dd77d745fc8d6; no voters:
I20260619 04:50:54.469523 25951 leader_election.cc:288] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:54.469666 25961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:54.470619 25961 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [term 1 LEADER]: Becoming Leader. State: Replica: dc76968c19b047e0ad4dd77d745fc8d6, State: Running, Role: LEADER
I20260619 04:50:54.471223 25961 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76968c19b047e0ad4dd77d745fc8d6" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39499 } }
I20260619 04:50:54.471689 25951 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:54.474630 25962 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc76968c19b047e0ad4dd77d745fc8d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76968c19b047e0ad4dd77d745fc8d6" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39499 } } }
I20260619 04:50:54.475076 25962 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:54.474972 25963 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc76968c19b047e0ad4dd77d745fc8d6. Latest consensus state: current_term: 1 leader_uuid: "dc76968c19b047e0ad4dd77d745fc8d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc76968c19b047e0ad4dd77d745fc8d6" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 39499 } } }
I20260619 04:50:54.475467 25963 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:54.482420 25969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:54.487730 25969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:54.496610 25969 catalog_manager.cc:1347] Generated new cluster ID: 361d9767b05c45eda7c30bca16a3c09c
I20260619 04:50:54.496742 25969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:54.536652 25969 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:54.538112 25969 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:54.547539 25969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc76968c19b047e0ad4dd77d745fc8d6: Generated new TSK 0
I20260619 04:50:54.548717 25969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:54.556959 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25884
W20260619 04:50:54.558180 25946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43071
W20260619 04:50:54.558938 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25884: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:54.559056 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 25884
W20260619 04:50:54.576033 25869 connection.cc:535] client connection to 127.3.243.190:39499 recv error: Network error: failed to read from TLS socket (remote: 127.3.243.190:39499): Connection reset by peer (error 104)
2026-06-19T04:50:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1891 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:54 dist-test-slave-2qp8 krb5kdc[25982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:54 dist-test-slave-2qp8 krb5kdc[25982](info): set up 2 sockets
Jun 19 04:50:54 dist-test-slave-2qp8 krb5kdc[25982](info): commencing operation
krb5kdc: starting...
W20260619 04:50:55.640684 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:55 dist-test-slave-2qp8 krb5kdc[25982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:50:55.880959 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:33077
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:42741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:33077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:56.203722 25995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:56.204073 25995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:56.204260 25995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:56.211500 25995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:56.211601 25995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:56.211647 25995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:56.211701 25995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:56.211737 25995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:56.221465 25995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:42741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:33077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:33077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:56.222749 25995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:56.224155 25995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:56.235875 26000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:56.235904 26001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:56.236387 26003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:56.236693 25995 server_base.cc:1034] running on GCE node
I20260619 04:50:56.237380 25995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:56.238695 25995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:56.239894 25995 hybrid_clock.cc:648] HybridClock initialized: now 1781844656239851 us; error 34 us; skew 500 ppm
I20260619 04:50:56.240304 25995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:56 dist-test-slave-2qp8 krb5kdc[25982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844656, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:56.248294 25995 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:50:56.249794 25995 webserver.cc:460] Webserver started at http://127.3.243.190:43393/ using document root <none> and password file <none>
I20260619 04:50:56.250476 25995 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:56.250641 25995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:56.250964 25995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:56.254082 25995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "64a9517e12b147c08f7b0b514bf17b9f"
format_stamp: "Formatted at 2026-06-19 04:50:56 on dist-test-slave-2qp8"
server_key: "9dbba02bd515cb49f23165ee9ccc6dcd"
server_key_iv: "9596683c5f42d7fc29d4714fbba8af71"
server_key_version: "encryptionkey@0"
I20260619 04:50:56.254827 25995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "64a9517e12b147c08f7b0b514bf17b9f"
format_stamp: "Formatted at 2026-06-19 04:50:56 on dist-test-slave-2qp8"
server_key: "9dbba02bd515cb49f23165ee9ccc6dcd"
server_key_iv: "9596683c5f42d7fc29d4714fbba8af71"
server_key_version: "encryptionkey@0"
I20260619 04:50:56.259945 25995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 04:50:56.263443 26010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:56.264834 25995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:50:56.265019 25995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "64a9517e12b147c08f7b0b514bf17b9f"
format_stamp: "Formatted at 2026-06-19 04:50:56 on dist-test-slave-2qp8"
server_key: "9dbba02bd515cb49f23165ee9ccc6dcd"
server_key_iv: "9596683c5f42d7fc29d4714fbba8af71"
server_key_version: "encryptionkey@0"
I20260619 04:50:56.265223 25995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:56.271034 25995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:56.274909 25995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:56.275192 25995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:56.293114 25995 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:33077
I20260619 04:50:56.293135 26061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:33077 every 8 connection(s)
I20260619 04:50:56.294642 25995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:56.298290 26062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:50:56.304484 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 25995
I20260619 04:50:56.304649 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:56.305078 4046 external_mini_cluster.cc:1428] Setting key b7918a01ff3fe163d81b4fc4b6e647e7
I20260619 04:50:56.308898 26062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f: Bootstrap starting.
I20260619 04:50:56.312539 26062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:56.312907 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:56.306875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60055 (local address 127.3.243.190:33077)
0619 04:50:56.307684 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:56.307695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:56.307735 (+ 40us) server_negotiation.cc:407] Connection header received
0619 04:50:56.309459 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:56.309481 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:56.309807 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:56.310119 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:56.311263 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:56.311272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:56.311284 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:56.311297 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:56.311353 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:56.311375 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:56.311533 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:56.311688 (+ 155us) server_negotiation.cc:299] Negotiation successful
0619 04:50:56.311885 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":396,"threads_started":1}
I20260619 04:50:56.313828 26062 log.cc:824] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:56.316758 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:56.316928 26062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f: No bootstrap required, opened a new log
I20260619 04:50:56.323736 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:56.321385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60729 (local address 127.3.243.190:33077)
0619 04:50:56.321606 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:56.321611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:56.321920 (+ 309us) server_negotiation.cc:407] Connection header received
0619 04:50:56.322068 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:56.322071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:56.322115 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:56.322200 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:56.323100 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:56.323104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:56.323107 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:50:56.323120 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:56.323158 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:50:56.323172 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:56.323271 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:56.323391 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 04:50:56.323440 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 04:50:56.325376 26062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a9517e12b147c08f7b0b514bf17b9f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33077 } }
I20260619 04:50:56.325726 26062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:56.325850 26062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a9517e12b147c08f7b0b514bf17b9f, State: Initialized, Role: FOLLOWER
I20260619 04:50:56.326471 26062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a9517e12b147c08f7b0b514bf17b9f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33077 } }
I20260619 04:50:56.326721 26062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:56.326867 26062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:56.327065 26062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:56.329241 26062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a9517e12b147c08f7b0b514bf17b9f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33077 } }
I20260619 04:50:56.329713 26062 leader_election.cc:302] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a9517e12b147c08f7b0b514bf17b9f; no voters:
I20260619 04:50:56.330250 26062 leader_election.cc:288] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:56.330411 26072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:56.331794 26072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [term 1 LEADER]: Becoming Leader. State: Replica: 64a9517e12b147c08f7b0b514bf17b9f, State: Running, Role: LEADER
I20260619 04:50:56.332299 26062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:56.332340 26072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a9517e12b147c08f7b0b514bf17b9f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33077 } }
I20260619 04:50:56.335330 26073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a9517e12b147c08f7b0b514bf17b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a9517e12b147c08f7b0b514bf17b9f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33077 } } }
I20260619 04:50:56.335331 26074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a9517e12b147c08f7b0b514bf17b9f. Latest consensus state: current_term: 1 leader_uuid: "64a9517e12b147c08f7b0b514bf17b9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a9517e12b147c08f7b0b514bf17b9f" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 33077 } } }
I20260619 04:50:56.335724 26073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:56.335768 26074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:56.340035 26078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:56.345414 26078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:56.353780 26078 catalog_manager.cc:1347] Generated new cluster ID: 4283a77b2acb428bb6fc130442e1de04
I20260619 04:50:56.353912 26078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:56.364233 26078 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:56.364645 26078 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:56.379310 26078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a9517e12b147c08f7b0b514bf17b9f: Generated new TSK 0
I20260619 04:50:56.380198 26078 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:56.390750 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25995
W20260619 04:50:56.391762 26057 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60055
W20260619 04:50:56.392504 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 25995: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:50:56.392652 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 25995
2026-06-19T04:50:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1833 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:56 dist-test-slave-2qp8 krb5kdc[26093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:56 dist-test-slave-2qp8 krb5kdc[26093](info): set up 2 sockets
Jun 19 04:50:56 dist-test-slave-2qp8 krb5kdc[26093](info): commencing operation
krb5kdc: starting...
W20260619 04:50:57.492856 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:57 dist-test-slave-2qp8 krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:50:57.746685 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45173
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:33011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:45173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:50:58.068899 26106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:50:58.069213 26106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:50:58.069370 26106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:50:58.076385 26106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:50:58.076488 26106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:50:58.076606 26106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:50:58.076701 26106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:50:58.076781 26106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:50:58.086548 26106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:33011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:45173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:50:58.087831 26106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:50:58.089260 26106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:50:58.101892 26111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:58.101861 26112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:50:58.102129 26114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:50:58.102983 26106 server_base.cc:1034] running on GCE node
I20260619 04:50:58.103787 26106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:50:58.105340 26106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:50:58.106686 26106 hybrid_clock.cc:648] HybridClock initialized: now 1781844658106632 us; error 36 us; skew 500 ppm
I20260619 04:50:58.107065 26106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:50:58 dist-test-slave-2qp8 krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844658, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:50:58.110071 26106 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:50:58.111495 26106 webserver.cc:460] Webserver started at http://127.3.243.190:36673/ using document root <none> and password file <none>
I20260619 04:50:58.112231 26106 fs_manager.cc:359] Metadata directory not provided
I20260619 04:50:58.112375 26106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:50:58.112676 26106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:50:58.115604 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "1532b28dff7b49dcb0e71699d57810d5"
format_stamp: "Formatted at 2026-06-19 04:50:58 on dist-test-slave-2qp8"
server_key: "426ddb2387ef987e157e687c1b5604bd"
server_key_iv: "61896eec12ab4a3d71636c07598d4d47"
server_key_version: "encryptionkey@0"
I20260619 04:50:58.116415 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "1532b28dff7b49dcb0e71699d57810d5"
format_stamp: "Formatted at 2026-06-19 04:50:58 on dist-test-slave-2qp8"
server_key: "426ddb2387ef987e157e687c1b5604bd"
server_key_iv: "61896eec12ab4a3d71636c07598d4d47"
server_key_version: "encryptionkey@0"
I20260619 04:50:58.121454 26106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 04:50:58.124841 26121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:50:58.126093 26106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:50:58.126276 26106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "1532b28dff7b49dcb0e71699d57810d5"
format_stamp: "Formatted at 2026-06-19 04:50:58 on dist-test-slave-2qp8"
server_key: "426ddb2387ef987e157e687c1b5604bd"
server_key_iv: "61896eec12ab4a3d71636c07598d4d47"
server_key_version: "encryptionkey@0"
I20260619 04:50:58.126463 26106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:50:58.163650 26106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:50:58.167760 26106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:50:58.168049 26106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:50:58.188550 26106 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:45173
I20260619 04:50:58.188572 26172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:45173 every 8 connection(s)
I20260619 04:50:58.190132 26106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:50:58.190707 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 26106
I20260619 04:50:58.190881 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:50:58.191502 4046 external_mini_cluster.cc:1428] Setting key 6847f109adc5b2543f544256317c2e97
I20260619 04:50:58.194514 26173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 04:50:58 dist-test-slave-2qp8 krb5kdc[26093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:50:58.207450 26173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5: Bootstrap starting.
I20260619 04:50:58.210527 26173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5: Neither blocks nor log segments found. Creating new log.
I20260619 04:50:58.211575 26173 log.cc:824] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:50:58.213900 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:58.193328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40597 (local address 127.3.243.190:45173)
0619 04:50:58.194093 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:58.194105 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:58.194142 (+ 37us) server_negotiation.cc:407] Connection header received
0619 04:50:58.195814 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:58.195835 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:58.196256 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:58.196560 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:58.198272 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:58.199577 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:58.201393 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:58.202066 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:58.202197 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:58.206858 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:58.206865 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:58.206880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:58.206918 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:58.209925 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:58.210607 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:58.210613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:58.210619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:58.210830 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:58.211206 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:58.211211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:58.211215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:58.211754 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:58.212077 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:58.212608 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 04:50:58.212994 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":344,"threads_started":1}
I20260619 04:50:58.214452 26173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5: No bootstrap required, opened a new log
I20260619 04:50:58.217777 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:50:58.223207 26173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1532b28dff7b49dcb0e71699d57810d5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45173 } }
I20260619 04:50:58.223582 26173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:50:58.223686 26173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1532b28dff7b49dcb0e71699d57810d5, State: Initialized, Role: FOLLOWER
I20260619 04:50:58.224347 26173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1532b28dff7b49dcb0e71699d57810d5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45173 } }
I20260619 04:50:58.224576 26173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:50:58.224718 26173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:50:58.224911 26173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:50:58.227103 26173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1532b28dff7b49dcb0e71699d57810d5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45173 } }
I20260619 04:50:58.227595 26173 leader_election.cc:302] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1532b28dff7b49dcb0e71699d57810d5; no voters:
I20260619 04:50:58.228296 26173 leader_election.cc:288] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:50:58.228442 26183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:50:58.230352 26173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:50:58.232801 26183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [term 1 LEADER]: Becoming Leader. State: Replica: 1532b28dff7b49dcb0e71699d57810d5, State: Running, Role: LEADER
I20260619 04:50:58.233390 26183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1532b28dff7b49dcb0e71699d57810d5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45173 } }
I20260619 04:50:58.236557 26185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1532b28dff7b49dcb0e71699d57810d5. Latest consensus state: current_term: 1 leader_uuid: "1532b28dff7b49dcb0e71699d57810d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1532b28dff7b49dcb0e71699d57810d5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45173 } } }
I20260619 04:50:58.236944 26185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:58.237376 26184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1532b28dff7b49dcb0e71699d57810d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1532b28dff7b49dcb0e71699d57810d5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45173 } } }
I20260619 04:50:58.237661 26184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:50:58.239756 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:50:58.222084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54959 (local address 127.3.243.190:45173)
0619 04:50:58.222270 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:50:58.222276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:50:58.222645 (+ 369us) server_negotiation.cc:407] Connection header received
0619 04:50:58.222846 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:50:58.222850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:50:58.222906 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:50:58.223000 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:50:58.224679 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:58.225518 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:58.226692 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:50:58.227162 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:50:58.227307 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:50:58.230649 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:50:58.230655 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:50:58.230660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:50:58.230695 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:50:58.235298 (+ 4603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:58.236262 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:58.236270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:58.236276 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:58.236534 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:50:58.237012 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:50:58.237019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:50:58.237023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:50:58.237451 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:50:58.237736 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 04:50:58.238997 (+ 1261us) server_negotiation.cc:299] Negotiation successful
0619 04:50:58.239303 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:50:58.241236 26191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:50:58.246448 26191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:50:58.255731 26191 catalog_manager.cc:1347] Generated new cluster ID: cc7a49553ba44bf18b2233a2940f9dac
I20260619 04:50:58.255879 26191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:50:58.287638 26191 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:50:58.289060 26191 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:50:58.295789 26191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1532b28dff7b49dcb0e71699d57810d5: Generated new TSK 0
I20260619 04:50:58.296612 26191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:50:58.303181 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26106
I20260619 04:50:58.304462 26168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:50:58.395640 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 26106
2026-06-19T04:50:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:50:58 dist-test-slave-2qp8 krb5kdc[26205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:50:58 dist-test-slave-2qp8 krb5kdc[26205](info): set up 2 sockets
Jun 19 04:50:58 dist-test-slave-2qp8 krb5kdc[26205](info): commencing operation
krb5kdc: starting...
W20260619 04:50:59.480724 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:50:59 dist-test-slave-2qp8 krb5kdc[26205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844659, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:50:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:50:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:50:59.733734 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:38015
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:46073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:38015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:51:00.073808 26218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:51:00.074229 26218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:51:00.074396 26218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:51:00.081933 26218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:51:00.082098 26218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:51:00.082209 26218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:51:00.082314 26218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:51:00.082387 26218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:51:00.092371 26218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:46073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:38015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:38015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:51:00.093643 26218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:51:00.094978 26218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:51:00.106621 26223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:00.107061 26224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:00.107088 26226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:51:00.108181 26218 server_base.cc:1034] running on GCE node
I20260619 04:51:00.109160 26218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:51:00.110628 26218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:51:00.111856 26218 hybrid_clock.cc:648] HybridClock initialized: now 1781844660111794 us; error 50 us; skew 500 ppm
I20260619 04:51:00.112267 26218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:51:00 dist-test-slave-2qp8 krb5kdc[26205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844660, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:51:00.115262 26218 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:51:00.116704 26218 webserver.cc:460] Webserver started at http://127.3.243.190:46673/ using document root <none> and password file <none>
I20260619 04:51:00.117390 26218 fs_manager.cc:359] Metadata directory not provided
I20260619 04:51:00.117529 26218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:51:00.117820 26218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:51:00.120771 26218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4"
format_stamp: "Formatted at 2026-06-19 04:51:00 on dist-test-slave-2qp8"
server_key: "9c38a52a717a395f8abb9f7d5c2d63e0"
server_key_iv: "dc534f5a7df3d0981d14730cbf0e4d51"
server_key_version: "encryptionkey@0"
I20260619 04:51:00.121557 26218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4"
format_stamp: "Formatted at 2026-06-19 04:51:00 on dist-test-slave-2qp8"
server_key: "9c38a52a717a395f8abb9f7d5c2d63e0"
server_key_iv: "dc534f5a7df3d0981d14730cbf0e4d51"
server_key_version: "encryptionkey@0"
I20260619 04:51:00.126600 26218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:51:00.129977 26233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:51:00.131361 26218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 04:51:00.131527 26218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4"
format_stamp: "Formatted at 2026-06-19 04:51:00 on dist-test-slave-2qp8"
server_key: "9c38a52a717a395f8abb9f7d5c2d63e0"
server_key_iv: "dc534f5a7df3d0981d14730cbf0e4d51"
server_key_version: "encryptionkey@0"
I20260619 04:51:00.131717 26218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:51:00.152343 26218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:51:00.156105 26218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:51:00.156420 26218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:51:00.174731 26218 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:38015
I20260619 04:51:00.174750 26284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:38015 every 8 connection(s)
I20260619 04:51:00.176103 26218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:51:00.178164 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 26218
I20260619 04:51:00.178347 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:51:00.178900 4046 external_mini_cluster.cc:1428] Setting key b6128f005b501375a091b557760749ca
I20260619 04:51:00.180498 26285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:51:00.195076 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:00.180760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49487 (local address 127.3.243.190:38015)
0619 04:51:00.181508 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:00.181519 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:00.181587 (+ 68us) server_negotiation.cc:407] Connection header received
0619 04:51:00.184265 (+ 2678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:00.184297 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:00.184952 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:00.185457 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:00.186040 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:00.188154 (+ 2114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:00.190020 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:00.191053 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:00.191231 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:00.192811 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:00.192819 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:00.192835 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:51:00.193413 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:00.193484 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:51:00.193514 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:00.193614 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:00.193846 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 04:51:00.194140 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":330,"threads_started":1}
I20260619 04:51:00.196046 26285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4: Bootstrap starting.
I20260619 04:51:00.198937 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:51:00.199801 26285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4: Neither blocks nor log segments found. Creating new log.
I20260619 04:51:00.200850 26285 log.cc:824] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4: Log is configured to *not* fsync() on all Append() calls
I20260619 04:51:00.204280 26285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4: No bootstrap required, opened a new log
I20260619 04:51:00.210217 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:00.203462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44717 (local address 127.3.243.190:38015)
0619 04:51:00.203680 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:00.203689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:00.204177 (+ 488us) server_negotiation.cc:407] Connection header received
0619 04:51:00.204442 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:00.204446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:00.204507 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:00.204581 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:00.205196 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:00.205893 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:00.207786 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:00.208303 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:00.208394 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:00.209489 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:00.209494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:00.209497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:51:00.209514 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:00.209553 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:51:00.209567 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:00.209643 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:00.209790 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 04:51:00.209858 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:51:00.213784 26285 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38015 } }
I20260619 04:51:00.214165 26285 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:51:00.214290 26285 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9403fb2feeb414aad1f24c1fc3aa3f4, State: Initialized, Role: FOLLOWER
I20260619 04:51:00.214869 26285 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38015 } }
I20260619 04:51:00.215086 26285 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:51:00.215233 26285 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:51:00.215400 26285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:51:00.217754 26285 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38015 } }
I20260619 04:51:00.218272 26285 leader_election.cc:302] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9403fb2feeb414aad1f24c1fc3aa3f4; no voters:
I20260619 04:51:00.218828 26285 leader_election.cc:288] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:51:00.218992 26295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:51:00.219798 26295 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [term 1 LEADER]: Becoming Leader. State: Replica: e9403fb2feeb414aad1f24c1fc3aa3f4, State: Running, Role: LEADER
I20260619 04:51:00.220400 26295 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38015 } }
I20260619 04:51:00.220983 26285 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:51:00.223615 26296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38015 } } }
I20260619 04:51:00.223694 26297 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9403fb2feeb414aad1f24c1fc3aa3f4. Latest consensus state: current_term: 1 leader_uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9403fb2feeb414aad1f24c1fc3aa3f4" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 38015 } } }
I20260619 04:51:00.224092 26297 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:00.224074 26296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4 [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:00.227272 26301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:51:00.233479 26301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:51:00.242906 26301 catalog_manager.cc:1347] Generated new cluster ID: 08598b660888435498dc01cef9bbd2fd
I20260619 04:51:00.243023 26301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:51:00.257162 26301 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:51:00.258431 26301 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:51:00.274274 26301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9403fb2feeb414aad1f24c1fc3aa3f4: Generated new TSK 0
I20260619 04:51:00.275098 26301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:51:00.285331 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26218
W20260619 04:51:00.286492 26280 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49487
W20260619 04:51:00.287209 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:51:00.287367 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 26218
2026-06-19T04:51:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1892 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:51:00 dist-test-slave-2qp8 krb5kdc[26316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:51:00 dist-test-slave-2qp8 krb5kdc[26316](info): set up 2 sockets
Jun 19 04:51:00 dist-test-slave-2qp8 krb5kdc[26316](info): commencing operation
krb5kdc: starting...
W20260619 04:51:01.372699 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:51:01 dist-test-slave-2qp8 krb5kdc[26316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:51:01.627908 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46729
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:37033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:46729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:51:01.948278 26329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:51:01.948565 26329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:51:01.948655 26329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:51:01.955511 26329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:51:01.955602 26329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:51:01.955646 26329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:51:01.955704 26329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:51:01.955745 26329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:51:01.965207 26329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:37033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:46729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:46729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:51:01.966352 26329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:51:01.967540 26329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:51:01.979022 26334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:01.979058 26337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:01.979146 26335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:51:01.979719 26329 server_base.cc:1034] running on GCE node
I20260619 04:51:01.980451 26329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:51:01.981721 26329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:51:01.982913 26329 hybrid_clock.cc:648] HybridClock initialized: now 1781844661982876 us; error 41 us; skew 500 ppm
I20260619 04:51:01.983230 26329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:51:01 dist-test-slave-2qp8 krb5kdc[26316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844661, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:51:01.989445 26329 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:51:01.990935 26329 webserver.cc:460] Webserver started at http://127.3.243.190:35855/ using document root <none> and password file <none>
I20260619 04:51:01.991586 26329 fs_manager.cc:359] Metadata directory not provided
I20260619 04:51:01.991711 26329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:51:01.992004 26329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:51:01.994856 26329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "5df248b9c933465592140623bd4fe785"
format_stamp: "Formatted at 2026-06-19 04:51:01 on dist-test-slave-2qp8"
server_key: "04d842209ad1899ff2ed54c8ba5c11bf"
server_key_iv: "b42a350e11b5f8db485cab1da571b0dd"
server_key_version: "encryptionkey@0"
I20260619 04:51:01.995623 26329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "5df248b9c933465592140623bd4fe785"
format_stamp: "Formatted at 2026-06-19 04:51:01 on dist-test-slave-2qp8"
server_key: "04d842209ad1899ff2ed54c8ba5c11bf"
server_key_iv: "b42a350e11b5f8db485cab1da571b0dd"
server_key_version: "encryptionkey@0"
I20260619 04:51:02.000617 26329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 04:51:02.003760 26344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:51:02.005112 26329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:51:02.005295 26329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "5df248b9c933465592140623bd4fe785"
format_stamp: "Formatted at 2026-06-19 04:51:01 on dist-test-slave-2qp8"
server_key: "04d842209ad1899ff2ed54c8ba5c11bf"
server_key_iv: "b42a350e11b5f8db485cab1da571b0dd"
server_key_version: "encryptionkey@0"
I20260619 04:51:02.005458 26329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:51:02.010934 26329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:51:02.014786 26329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:51:02.015057 26329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:51:02.033946 26329 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:46729
I20260619 04:51:02.033977 26395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:46729 every 8 connection(s)
I20260619 04:51:02.035269 26329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:51:02.038947 26396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:51:02.041003 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 26329
I20260619 04:51:02.041178 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:51:02.041740 4046 external_mini_cluster.cc:1428] Setting key 2ef2680ab0fba3b5d8c77ee290763b95
I20260619 04:51:02.049001 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:02.043459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33953 (local address 127.3.243.190:46729)
0619 04:51:02.044276 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:02.044288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:02.044329 (+ 41us) server_negotiation.cc:407] Connection header received
0619 04:51:02.045861 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:02.045884 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:02.046204 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:02.046514 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:02.047526 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:02.047532 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:02.047545 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:51:02.047557 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:02.047613 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:51:02.047631 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:02.047792 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:02.047932 (+ 140us) server_negotiation.cc:299] Negotiation successful
0619 04:51:02.048161 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":393,"threads_started":1}
I20260619 04:51:02.052049 26396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785: Bootstrap starting.
I20260619 04:51:02.052510 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:51:02.055536 26396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785: Neither blocks nor log segments found. Creating new log.
I20260619 04:51:02.056818 26396 log.cc:824] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785: Log is configured to *not* fsync() on all Append() calls
I20260619 04:51:02.059814 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:02.057246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42385 (local address 127.3.243.190:46729)
0619 04:51:02.057380 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:02.057383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:02.058008 (+ 625us) server_negotiation.cc:407] Connection header received
0619 04:51:02.058173 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:02.058180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:02.058236 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:02.058332 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:02.059173 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:02.059180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:02.059184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 04:51:02.059199 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:02.059241 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 04:51:02.059258 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:02.059393 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:02.059484 (+ 91us) server_negotiation.cc:299] Negotiation successful
0619 04:51:02.059538 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 04:51:02.059813 26396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785: No bootstrap required, opened a new log
I20260619 04:51:02.067483 26396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df248b9c933465592140623bd4fe785" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46729 } }
I20260619 04:51:02.067785 26396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 04:51:02.067860 26396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5df248b9c933465592140623bd4fe785, State: Initialized, Role: FOLLOWER
I20260619 04:51:02.068357 26396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df248b9c933465592140623bd4fe785" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46729 } }
I20260619 04:51:02.068549 26396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 04:51:02.068641 26396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 04:51:02.068751 26396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:51:02.071027 26396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df248b9c933465592140623bd4fe785" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46729 } }
I20260619 04:51:02.071552 26396 leader_election.cc:302] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5df248b9c933465592140623bd4fe785; no voters:
I20260619 04:51:02.072093 26396 leader_election.cc:288] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 04:51:02.072337 26406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:51:02.073009 26406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [term 1 LEADER]: Becoming Leader. State: Replica: 5df248b9c933465592140623bd4fe785, State: Running, Role: LEADER
I20260619 04:51:02.073532 26406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df248b9c933465592140623bd4fe785" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46729 } }
I20260619 04:51:02.074154 26396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:51:02.076507 26408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5df248b9c933465592140623bd4fe785. Latest consensus state: current_term: 1 leader_uuid: "5df248b9c933465592140623bd4fe785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df248b9c933465592140623bd4fe785" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46729 } } }
I20260619 04:51:02.077013 26408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:02.076472 26407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5df248b9c933465592140623bd4fe785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5df248b9c933465592140623bd4fe785" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 46729 } } }
I20260619 04:51:02.077832 26407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785 [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:02.078841 26415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:51:02.082818 26415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 04:51:02.090974 26415 catalog_manager.cc:1347] Generated new cluster ID: 89a6600e57e84355908703ac99a073fb
I20260619 04:51:02.091091 26415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:51:02.107803 26415 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:51:02.108224 26415 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:51:02.119421 26415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5df248b9c933465592140623bd4fe785: Generated new TSK 0
I20260619 04:51:02.120297 26415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:51:02.127642 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26329
W20260619 04:51:02.128674 26391 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33953
W20260619 04:51:02.129346 4046 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26329: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 04:51:02.129499 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 26329
2026-06-19T04:51:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1840 ms)
[----------] 18 tests from AuthTokenIssuingTest (34562 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:51:02 dist-test-slave-2qp8 krb5kdc[26427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:51:02 dist-test-slave-2qp8 krb5kdc[26427](info): set up 2 sockets
Jun 19 04:51:02 dist-test-slave-2qp8 krb5kdc[26427](info): commencing operation
krb5kdc: starting...
W20260619 04:51:03.276656 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:51:03 dist-test-slave-2qp8 krb5kdc[26427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:51:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:51:03.527784 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44063
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44063,127.3.243.189:40021,127.3.243.188:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:51:03.850996 26440 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:51:03.851281 26440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:51:03.851440 26440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:51:03.851593 26440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:51:03.858774 26440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:51:03.858853 26440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:51:03.858950 26440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:51:03.859026 26440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:51:03.859139 26440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:51:03.868999 26440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44063,127.3.243.189:40021,127.3.243.188:33109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:44063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:51:03.870268 26440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:51:03.871570 26440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:51:03.883275 26445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:03.883634 26446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:03.883639 26448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:51:03.884469 26440 server_base.cc:1034] running on GCE node
I20260619 04:51:03.885188 26440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:51:03.886389 26440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:51:03.887593 26440 hybrid_clock.cc:648] HybridClock initialized: now 1781844663887517 us; error 50 us; skew 500 ppm
I20260619 04:51:03.887857 26440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:51:03 dist-test-slave-2qp8 krb5kdc[26427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844663, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:51:03.891049 26440 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:51:03.892563 26440 webserver.cc:460] Webserver started at http://127.3.243.190:34285/ using document root <none> and password file <none>
I20260619 04:51:03.893287 26440 fs_manager.cc:359] Metadata directory not provided
I20260619 04:51:03.893419 26440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:51:03.893755 26440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:51:03.896703 26440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "eba364bdc478484c9b1cac5e425313b5"
format_stamp: "Formatted at 2026-06-19 04:51:03 on dist-test-slave-2qp8"
server_key: "b98b682935b2856e8ed73122db0a711e"
server_key_iv: "deaa323a2dd648c4dcf28181b98815d8"
server_key_version: "encryptionkey@0"
I20260619 04:51:03.897461 26440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "eba364bdc478484c9b1cac5e425313b5"
format_stamp: "Formatted at 2026-06-19 04:51:03 on dist-test-slave-2qp8"
server_key: "b98b682935b2856e8ed73122db0a711e"
server_key_iv: "deaa323a2dd648c4dcf28181b98815d8"
server_key_version: "encryptionkey@0"
I20260619 04:51:03.902480 26440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:51:03.905876 26455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:51:03.907123 26440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 04:51:03.907323 26440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "eba364bdc478484c9b1cac5e425313b5"
format_stamp: "Formatted at 2026-06-19 04:51:03 on dist-test-slave-2qp8"
server_key: "b98b682935b2856e8ed73122db0a711e"
server_key_iv: "deaa323a2dd648c4dcf28181b98815d8"
server_key_version: "encryptionkey@0"
I20260619 04:51:03.907542 26440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:51:03.935128 26440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:51:03.939160 26440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:51:03.939459 26440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:51:03.958390 26440 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:44063
I20260619 04:51:03.958410 26506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:44063 every 8 connection(s)
I20260619 04:51:03.959733 26440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:51:03.962880 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 26440
I20260619 04:51:03.963106 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:51:03.963666 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:51:03.963755 4046 external_mini_cluster.cc:1428] Setting key 93a142031f98af44a4fd1b08f1205b34
I20260619 04:51:03.970610 26507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:03 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844663, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.189@KRBTEST.COM" created.
I20260619 04:51:03.996270 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:03.973987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51234 (local address 127.3.243.190:44063)
0619 04:51:03.975952 (+ 1965us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:03.975962 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:03.976004 (+ 42us) server_negotiation.cc:407] Connection header received
0619 04:51:03.978131 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:03.978136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:03.978387 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:03.978553 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:03.980256 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:03.981561 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:03.983771 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:03.984488 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:03.984628 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:03.988756 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:03.988764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:03.988775 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:03.988808 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:03.991915 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:03.992729 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:03.992741 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:03.992747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:03.992929 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:03.993396 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:03.993403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:03.993409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:03.994097 (+ 688us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:03.994291 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:03.994938 (+ 647us) server_negotiation.cc:299] Negotiation successful
0619 04:51:03.995365 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":1771,"thread_start_us":1495,"threads_started":1}
I20260619 04:51:03.996322 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:03.973987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44063 (local address 127.0.0.1:51234)
0619 04:51:03.974919 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:03.974977 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:03.977744 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:03.978632 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:03.978641 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:03.979456 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:03.979960 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:03.979987 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:03.981735 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:03.981743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:03.983624 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:03.983640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:03.984739 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:03.984749 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:03.984980 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:03.985718 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:03.985755 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:03.988523 (+ 2768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:03.992054 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:03.992064 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:03.992073 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:03.992590 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:03.993061 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:03.993065 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:03.993067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:03.993230 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:03.994233 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:03.994240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:03.994469 (+ 229us) client_negotiation.cc:769] Sending connection context
0619 04:51:03.994705 (+ 236us) client_negotiation.cc:240] Negotiation successful
0619 04:51:03.995150 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":376,"threads_started":1}
I20260619 04:51:03.999482 26507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:51:04.000660 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.000232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46862)
0619 04:51:04.000437 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.000549 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 04:51:04.000967 26458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
I20260619 04:51:04.002578 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.002097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46864)
0619 04:51:04.002313 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.002415 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 04:51:04.002794 26507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:40021: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab.
Entry for principal kudu/127.3.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab.
I20260619 04:51:04.023064 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.189:40021
--webserver_interface=127.3.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab
--principal=kudu/127.3.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44063,127.3.243.189:40021,127.3.243.188:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:51:04.052165 26507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } attempt: 1
I20260619 04:51:04.053400 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.052872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46868)
0619 04:51:04.053084 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.053213 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 04:51:04.054355 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.053989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46882)
0619 04:51:04.054127 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.054203 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 04:51:04.054530 26507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:40021: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
I20260619 04:51:04.122849 26507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } attempt: 2
I20260619 04:51:04.124634 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.123724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46884)
0619 04:51:04.124348 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.124456 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428}
I20260619 04:51:04.125765 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.125330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46888)
0619 04:51:04.125519 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.125617 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 04:51:04.125964 26507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:40021: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
I20260619 04:51:04.217239 26507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } attempt: 3
I20260619 04:51:04.218626 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.218006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46900)
0619 04:51:04.218284 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.218420 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260619 04:51:04.219683 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.219287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46904)
0619 04:51:04.219446 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.219531 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 04:51:04.219866 26507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:40021: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
W20260619 04:51:04.359175 26514 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:51:04.359504 26514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:51:04.359577 26514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:51:04.359658 26514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 04:51:04.363126 26507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } attempt: 4
I20260619 04:51:04.364494 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.363789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46906)
0619 04:51:04.364179 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.364320 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260619 04:51:04.365543 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.365181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46908)
0619 04:51:04.365308 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.365398 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260619 04:51:04.365795 26507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:40021: Network error: Client connection negotiation failed: client connection to 127.3.243.189:40021: connect: Connection refused (error 111)
W20260619 04:51:04.367974 26514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:51:04.368050 26514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:51:04.368167 26514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:51:04.368240 26514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:51:04.368304 26514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:51:04.378113 26514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44063,127.3.243.189:40021,127.3.243.188:33109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab
--principal=kudu/127.3.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.189:40021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:51:04.379383 26514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:51:04.380712 26514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:51:04.392336 26519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:04.392457 26520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:04.392365 26522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:51:04.393630 26514 server_base.cc:1034] running on GCE node
I20260619 04:51:04.394487 26514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:51:04.396036 26514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:51:04.397311 26514 hybrid_clock.cc:648] HybridClock initialized: now 1781844664397248 us; error 29 us; skew 500 ppm
I20260619 04:51:04.397620 26514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:51:04 dist-test-slave-2qp8 krb5kdc[26427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844664, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:51:04.400823 26514 init.cc:377] Logged in from keytab as kudu/127.3.243.189@KRBTEST.COM (short username kudu)
I20260619 04:51:04.402367 26514 webserver.cc:460] Webserver started at http://127.3.243.189:33609/ using document root <none> and password file <none>
I20260619 04:51:04.403026 26514 fs_manager.cc:359] Metadata directory not provided
I20260619 04:51:04.403110 26514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:51:04.403380 26514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:51:04.406665 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/data/instance:
uuid: "022b1c71c06749249aa3e8f529db63aa"
format_stamp: "Formatted at 2026-06-19 04:51:04 on dist-test-slave-2qp8"
server_key: "94f450e9c1cef48b4c9a76db6a94bf9d"
server_key_iv: "91efc341f497bee67e5a4bacdbbc86a3"
server_key_version: "encryptionkey@0"
I20260619 04:51:04.407428 26514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/wal/instance:
uuid: "022b1c71c06749249aa3e8f529db63aa"
format_stamp: "Formatted at 2026-06-19 04:51:04 on dist-test-slave-2qp8"
server_key: "94f450e9c1cef48b4c9a76db6a94bf9d"
server_key_iv: "91efc341f497bee67e5a4bacdbbc86a3"
server_key_version: "encryptionkey@0"
I20260619 04:51:04.412739 26514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 04:51:04.415901 26529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:51:04.417369 26514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 04:51:04.417507 26514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/wal
uuid: "022b1c71c06749249aa3e8f529db63aa"
format_stamp: "Formatted at 2026-06-19 04:51:04 on dist-test-slave-2qp8"
server_key: "94f450e9c1cef48b4c9a76db6a94bf9d"
server_key_iv: "91efc341f497bee67e5a4bacdbbc86a3"
server_key_version: "encryptionkey@0"
I20260619 04:51:04.417685 26514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:51:04.451020 26514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:51:04.454926 26514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:51:04.455145 26514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:51:04.473986 26514 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.189:40021
I20260619 04:51:04.474007 26580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.189:40021 every 8 connection(s)
I20260619 04:51:04.475380 26514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
I20260619 04:51:04.477928 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 26514
I20260619 04:51:04.478142 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-1/wal/instance
I20260619 04:51:04.478633 4046 external_mini_cluster.cc:1428] Setting key bede7ac3ebe4dea166b05cf140be95b7
I20260619 04:51:04.479352 26581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:51:04.485888 26581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:04 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844664, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.188@KRBTEST.COM" created.
I20260619 04:51:04.508507 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.488980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.3.243.190:44063)
0619 04:51:04.489198 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:04.489204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:04.490370 (+ 1166us) server_negotiation.cc:407] Connection header received
0619 04:51:04.492298 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:04.492306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:04.492403 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:04.492523 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:04.494172 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.495312 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.497078 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.497890 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.498037 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:04.501876 (+ 3839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:04.501883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:04.501889 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:04.501942 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:04.505331 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:04.506117 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:04.506122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:04.506125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:04.506262 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:04.506687 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:04.506694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:04.506702 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:04.506977 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:04.507143 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:04.507887 (+ 744us) server_negotiation.cc:299] Negotiation successful
0619 04:51:04.508162 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 04:51:04.509627 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.489366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44063 (local address 127.0.0.1:51248)
0619 04:51:04.490176 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.490222 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:04.491975 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:04.492748 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:04.492757 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:04.493409 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:04.493982 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:04.494000 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.495491 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.495495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:04.496916 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:04.496928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.498118 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.498123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:04.498340 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:04.499070 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:04.499092 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:04.501540 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:04.505487 (+ 3947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:04.505498 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:04.505517 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:04.505976 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:04.506377 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:04.506382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:04.506386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:04.506567 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:04.507120 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:04.507128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:04.507498 (+ 370us) client_negotiation.cc:769] Sending connection context
0619 04:51:04.507788 (+ 290us) client_negotiation.cc:240] Negotiation successful
0619 04:51:04.508375 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":361,"threads_started":1}
I20260619 04:51:04.511530 26581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:04 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844664, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:51:04.527352 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.512570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46924)
0619 04:51:04.512804 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.512829 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:04.512947 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:04.514083 (+ 1136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:04.514086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:04.514452 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:04.514591 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:04.514599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.516779 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.516783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:04.517623 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:04.517634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.518987 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.518991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:04.519187 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:04.519800 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:04.519816 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:04.521885 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:04.525050 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:04.525058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:04.525062 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:04.525419 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:04.525783 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:04.525790 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:04.525793 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:04.525930 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:04.526497 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:04.526501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:04.526634 (+ 133us) client_negotiation.cc:769] Sending connection context
0619 04:51:04.526803 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 04:51:04.527075 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":45}
I20260619 04:51:04.527918 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.512838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.3.243.189:40021)
0619 04:51:04.513560 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:04.513567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:04.513591 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:51:04.513634 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:04.513640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:04.513883 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:04.514082 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:04.514766 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.516633 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.517761 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.518755 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.519026 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:04.522097 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:04.522108 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:04.522121 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:04.522151 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:04.524918 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:04.525546 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:04.525571 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:04.525574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:04.525678 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:04.526029 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:04.526036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:04.526039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:04.526381 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:04.526568 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:04.527019 (+ 451us) server_negotiation.cc:299] Negotiation successful
0619 04:51:04.527318 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":576,"thread_start_us":398,"threads_started":1}
I20260619 04:51:04.530545 26581 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:51:04.531752 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.531338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50406)
0619 04:51:04.531519 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.531611 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 04:51:04.532099 26531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
I20260619 04:51:04.533955 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.533655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50420)
0619 04:51:04.533784 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.533863 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 04:51:04.534186 26581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33109: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab.
Entry for principal kudu/127.3.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab.
I20260619 04:51:04.540036 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.188:33109
--webserver_interface=127.3.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab
--principal=kudu/127.3.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:44063,127.3.243.189:40021,127.3.243.188:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:51:04.583537 26581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } attempt: 1
I20260619 04:51:04.584779 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.584333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50422)
0619 04:51:04.584528 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.584634 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 04:51:04.586061 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.585530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50426)
0619 04:51:04.585789 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.585902 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260619 04:51:04.586436 26581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33109: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
I20260619 04:51:04.654760 26581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } attempt: 2
I20260619 04:51:04.656004 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.655529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50430)
0619 04:51:04.655725 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.655819 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 04:51:04.657002 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.656664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50436)
0619 04:51:04.656806 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.656884 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 04:51:04.657196 26581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33109: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
I20260619 04:51:04.665042 26507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } attempt: 5
Jun 19 04:51:04 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844663, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:51:04.679755 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.665670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46932)
0619 04:51:04.665865 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.665894 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:04.666001 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:04.666439 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:04.666442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:04.666794 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:04.666990 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:04.666998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.668380 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.668384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:04.669233 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:04.669243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.670166 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.670170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:04.670324 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:04.670950 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:04.670967 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:04.673186 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:04.677362 (+ 4176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:04.677366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:04.677369 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:04.677742 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:04.678149 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:04.678160 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:04.678164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:04.678336 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:04.678975 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:04.678979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:04.679121 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 04:51:04.679250 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 04:51:04.679498 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 04:51:04.679922 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.665769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.3.243.189:40021)
0619 04:51:04.665995 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:04.666003 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:04.666031 (+ 28us) server_negotiation.cc:407] Connection header received
0619 04:51:04.666107 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:04.666114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:04.666210 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:04.666400 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:04.667142 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.668246 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.669379 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.670024 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.670151 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:04.673402 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:04.673412 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:04.673420 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:04.673470 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:04.677195 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:04.677860 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:04.677866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:04.677872 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:04.678020 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:04.678452 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:04.678457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:04.678460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:04.678824 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:04.678975 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:04.679395 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 04:51:04.679618 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 04:51:04.681070 26507 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:51:04.682062 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.681691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50452)
0619 04:51:04.681854 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.681933 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 04:51:04.683050 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.682612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50462)
0619 04:51:04.682843 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.682943 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 04:51:04.683302 26507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33109: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
I20260619 04:51:04.734570 26507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } attempt: 1
I20260619 04:51:04.735853 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.735339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50466)
0619 04:51:04.735562 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.735663 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 04:51:04.736951 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.736617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50480)
0619 04:51:04.736757 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.736830 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 04:51:04.737282 26507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33109: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
I20260619 04:51:04.748406 26581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } attempt: 3
I20260619 04:51:04.749522 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.749066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50496)
0619 04:51:04.749263 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.749364 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 04:51:04.750655 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.750214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50498)
0619 04:51:04.750482 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.750556 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260619 04:51:04.750844 26581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33109: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
I20260619 04:51:04.805527 26507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } attempt: 2
I20260619 04:51:04.806749 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.806245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50510)
0619 04:51:04.806507 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.806592 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260619 04:51:04.807772 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.807422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50520)
0619 04:51:04.807589 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.807664 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 04:51:04.807982 26507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33109: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
W20260619 04:51:04.879819 26588 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:51:04.880275 26588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:51:04.880407 26588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:51:04.880543 26588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:51:04.887791 26588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:51:04.887946 26588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:51:04.888070 26588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:51:04.888180 26588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:51:04.888284 26588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:51:04.894068 26581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } attempt: 4
I20260619 04:51:04.895251 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.894692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50536)
0619 04:51:04.894953 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.895062 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 04:51:04.896353 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.895977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50546)
0619 04:51:04.896139 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.896224 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260619 04:51:04.896584 26581 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33109: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
I20260619 04:51:04.899619 26588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:44063,127.3.243.189:40021,127.3.243.188:33109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab
--principal=kudu/127.3.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.188:33109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:51:04.901031 26588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:51:04.902412 26588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 04:51:04.914152 26507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } attempt: 3
W20260619 04:51:04.914393 26594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:04.914427 26596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:04.914480 26593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:51:04.915532 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.914800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50560)
0619 04:51:04.915236 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.915339 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
I20260619 04:51:04.916765 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.916313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50564)
0619 04:51:04.916553 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:04.916630 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260619 04:51:04.916936 26588 server_base.cc:1034] running on GCE node
W20260619 04:51:04.917049 26507 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33109: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33109: connect: Connection refused (error 111)
I20260619 04:51:04.917793 26588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:51:04.919175 26588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:51:04.920449 26588 hybrid_clock.cc:648] HybridClock initialized: now 1781844664920383 us; error 39 us; skew 500 ppm
I20260619 04:51:04.920814 26588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:51:04 dist-test-slave-2qp8 krb5kdc[26427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844664, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:51:04.923981 26588 init.cc:377] Logged in from keytab as kudu/127.3.243.188@KRBTEST.COM (short username kudu)
I20260619 04:51:04.925565 26588 webserver.cc:460] Webserver started at http://127.3.243.188:34081/ using document root <none> and password file <none>
I20260619 04:51:04.926255 26588 fs_manager.cc:359] Metadata directory not provided
I20260619 04:51:04.926400 26588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:51:04.926717 26588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:51:04.929728 26588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/data/instance:
uuid: "cc09cee1327343eead6b474c5aafbbd7"
format_stamp: "Formatted at 2026-06-19 04:51:04 on dist-test-slave-2qp8"
server_key: "941176d49d60b165b69b9b1164b2c162"
server_key_iv: "c2611905d653752a79e46bec9bdd3113"
server_key_version: "encryptionkey@0"
I20260619 04:51:04.930517 26588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/wal/instance:
uuid: "cc09cee1327343eead6b474c5aafbbd7"
format_stamp: "Formatted at 2026-06-19 04:51:04 on dist-test-slave-2qp8"
server_key: "941176d49d60b165b69b9b1164b2c162"
server_key_iv: "c2611905d653752a79e46bec9bdd3113"
server_key_version: "encryptionkey@0"
I20260619 04:51:04.935660 26588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 04:51:04.939553 26603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:51:04.940941 26588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 04:51:04.941138 26588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/wal
uuid: "cc09cee1327343eead6b474c5aafbbd7"
format_stamp: "Formatted at 2026-06-19 04:51:04 on dist-test-slave-2qp8"
server_key: "941176d49d60b165b69b9b1164b2c162"
server_key_iv: "c2611905d653752a79e46bec9bdd3113"
server_key_version: "encryptionkey@0"
I20260619 04:51:04.941352 26588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:51:04.961032 26588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:51:04.965343 26588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:51:04.965628 26588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:51:04.984462 26588 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.188:33109
I20260619 04:51:04.984474 26654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.188:33109 every 8 connection(s)
I20260619 04:51:04.985803 26588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
I20260619 04:51:04.986506 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 26588
I20260619 04:51:04.986699 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781844312210132-4046-0/minicluster-data/master-2/wal/instance
I20260619 04:51:04.987191 4046 external_mini_cluster.cc:1428] Setting key be3b5cfeb74a9b4f9cb1b13b4e98eb48
I20260619 04:51:04.990351 26655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:51:04.999155 26655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:04 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
Jun 19 04:51:05 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844664, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:51:05.012454 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:04.989170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.3.243.188:33109)
0619 04:51:04.989925 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:04.989941 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:04.989993 (+ 52us) server_negotiation.cc:407] Connection header received
0619 04:51:04.992251 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:04.992282 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:04.992805 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:04.993252 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:04.994566 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.996662 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.997967 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:04.998983 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:04.999181 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.003758 (+ 4577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.003770 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.003790 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.003873 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.007537 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.008454 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.008481 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.008491 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.008706 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.009119 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.009123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.009125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.009956 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.010259 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.010733 (+ 474us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.011134 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"spinlock_wait_cycles":29312,"thread_start_us":313,"threads_started":1}
I20260619 04:51:05.018347 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.001862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:44063 (local address 127.0.0.1:51250)
0619 04:51:05.002731 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:05.002774 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:05.002928 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:05.003360 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:05.003365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:05.004294 (+ 929us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:05.004659 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.004669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.005849 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.005855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.007576 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.007590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.008591 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.008599 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.008785 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.009366 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:05.009388 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:05.012019 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:05.015518 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.015530 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.015536 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.016149 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.016605 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.016611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.016614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.016769 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.017329 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:05.017339 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:05.017567 (+ 228us) client_negotiation.cc:769] Sending connection context
0619 04:51:05.017755 (+ 188us) client_negotiation.cc:240] Negotiation successful
0619 04:51:05.018055 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":457,"threads_started":1}
I20260619 04:51:05.018498 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.001786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.3.243.190:44063)
0619 04:51:05.001972 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.001996 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.002894 (+ 898us) server_negotiation.cc:407] Connection header received
0619 04:51:05.003091 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.003097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.003185 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.003300 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.004839 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.005695 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.007707 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.008407 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.008524 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.012252 (+ 3728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.012259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.012263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.012313 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.015390 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.016248 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.016259 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.016263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.016450 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.016886 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.016889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.016891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.017163 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.017297 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.017835 (+ 538us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.018054 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:51:05.019793 26655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:05 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
Jun 19 04:51:05 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844664, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:51:05.031289 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.015618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.3.243.190:44063)
0619 04:51:05.016271 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.016275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.016294 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:51:05.016360 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.016364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.016430 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.016510 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.018054 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.019254 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.020857 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.021476 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.021607 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.025602 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.025610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.025614 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.025652 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.028544 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.029371 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.029375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.029377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.029498 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.029887 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.029893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.029898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.030229 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.030446 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.030782 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.030960 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":329,"threads_started":1}
I20260619 04:51:05.034431 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.020650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.3.243.189:40021)
0619 04:51:05.020852 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.020858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.020875 (+ 17us) server_negotiation.cc:407] Connection header received
0619 04:51:05.021058 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.021063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.021142 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.021241 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.021920 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.022764 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.023870 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.024510 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.024650 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.028071 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.028077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.028083 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.028163 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.031504 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.032293 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.032297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.032299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.032485 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.032883 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.032888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.032891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.033228 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.033374 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.033986 (+ 612us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.034140 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 04:51:05.034777 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.020648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:40021 (local address 127.0.0.1:46934)
0619 04:51:05.020787 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:05.020807 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:05.020928 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:05.021264 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:05.021267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:05.021568 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:05.021780 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.021792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.022906 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.022914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.023731 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.023741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.024714 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.024723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.024920 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.025514 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:05.025538 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:05.027874 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:05.031662 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.031666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.031668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.032176 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.032609 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.032614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.032618 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.032766 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.033441 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:05.033448 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:05.033683 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 04:51:05.033888 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 04:51:05.034274 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 04:51:05.036036 26655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:05 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844664, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
Jun 19 04:51:05 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:51:05.049942 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.036827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50580 (local address 127.3.243.188:33109)
0619 04:51:05.036999 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.037004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.037022 (+ 18us) server_negotiation.cc:407] Connection header received
0619 04:51:05.037150 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.037155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.037219 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.037304 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.037967 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.039013 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.040093 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.040516 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.040654 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.043672 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.043685 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.043688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.043713 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.047251 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.047846 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.047852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.047856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.048005 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.048486 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.048490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.048492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.048741 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.048904 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.049258 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.049537 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":70}
I20260619 04:51:05.050266 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.036788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50580)
0619 04:51:05.036946 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:05.036966 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:05.037055 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:05.037354 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:05.037358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:05.037705 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:05.037832 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.037840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.039139 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.039142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.039956 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.039967 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.040668 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.040674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.040876 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.041413 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:05.041428 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:05.043508 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:05.047378 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.047382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.047385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.047719 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.048173 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.048197 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.048203 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.048371 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.048878 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:05.048883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:05.049008 (+ 125us) client_negotiation.cc:769] Sending connection context
0619 04:51:05.049165 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 04:51:05.049730 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":186}
I20260619 04:51:05.051183 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.033281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.3.243.189:40021)
0619 04:51:05.034673 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.034679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.034694 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:51:05.034738 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.034743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.034815 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.034926 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.036560 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.037600 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.039362 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.039779 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.039929 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.044557 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.044567 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.044573 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.044614 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.048102 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.048869 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.048874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.048877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.049025 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.049450 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.049456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.049459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.049791 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.050075 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.050638 (+ 563us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.050854 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1297,"thread_start_us":1822,"threads_started":1}
I20260619 04:51:05.055897 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:51:05.056425 26655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7: Bootstrap starting.
I20260619 04:51:05.060232 26655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7: Neither blocks nor log segments found. Creating new log.
I20260619 04:51:05.061257 26655 log.cc:824] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7: Log is configured to *not* fsync() on all Append() calls
I20260619 04:51:05.059286 26482 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eba364bdc478484c9b1cac5e425313b5"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51254
I20260619 04:51:05.063951 26655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7: No bootstrap required, opened a new log
I20260619 04:51:05.070815 26655 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } }
I20260619 04:51:05.071154 26655 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc09cee1327343eead6b474c5aafbbd7, State: Initialized, Role: FOLLOWER
I20260619 04:51:05.071614 26655 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } }
I20260619 04:51:05.072623 26663 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.073014 26663 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:05.073717 26655 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 04:51:05.083323 26674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:51:05.083494 26674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:51:05.080982 26588 master_runner.cc:186] Error getting master registration for 127.3.243.190:44063: OK, instance_id { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" instance_seqno: 1781844663953253 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:51:05.092298 26588 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:51:05.094303 26507 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } attempt: 4
Jun 19 04:51:05 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844663, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
I20260619 04:51:05.111362 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.095037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50596)
0619 04:51:05.095246 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:05.095268 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:05.095416 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:05.095805 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:05.095810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:05.096343 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:05.096568 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.096579 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.098092 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.098098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.099606 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.099622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.100717 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.100723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.100935 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.101635 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:05.101662 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:05.104649 (+ 2987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:05.108701 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.108706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.108711 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.109074 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.109445 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.109466 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.109471 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.109637 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.110251 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:05.110256 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:05.110481 (+ 225us) client_negotiation.cc:769] Sending connection context
0619 04:51:05.110629 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 04:51:05.110918 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 04:51:05.111389 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.095126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.3.243.188:33109)
0619 04:51:05.095283 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.095291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.095376 (+ 85us) server_negotiation.cc:407] Connection header received
0619 04:51:05.095552 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.095558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.095640 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.095748 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.096767 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.097963 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.099751 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.100559 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.100699 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.104893 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.104899 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.104904 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.104944 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.108564 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.109200 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.109207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.109211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.109353 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.109740 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.109745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.109748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.110122 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.110295 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.110809 (+ 514us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.111029 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:51:05.116241 26507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5: Bootstrap starting.
I20260619 04:51:05.120357 26507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5: Neither blocks nor log segments found. Creating new log.
I20260619 04:51:05.121685 26507 log.cc:824] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5: Log is configured to *not* fsync() on all Append() calls
I20260619 04:51:05.125138 26507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5: No bootstrap required, opened a new log
I20260619 04:51:05.132913 26507 raft_consensus.cc:348] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } }
I20260619 04:51:05.133337 26507 raft_consensus.cc:729] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eba364bdc478484c9b1cac5e425313b5, State: Initialized, Role: FOLLOWER
I20260619 04:51:05.133836 26507 consensus_queue.cc:260] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } }
I20260619 04:51:05.135084 26677 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.135665 26677 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:05.136400 26507 sys_catalog.cc:564] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:51:05.136475 26482 raft_consensus.cc:482] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 04:51:05.136649 26482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:51:05.140775 26482 raft_consensus.cc:504] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } }
I20260619 04:51:05.143064 26482 leader_election.cc:288] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [CANDIDATE]: Term 1 election: Requested vote from peers 022b1c71c06749249aa3e8f529db63aa (127.3.243.189:40021), cc09cee1327343eead6b474c5aafbbd7 (127.3.243.188:33109)
I20260619 04:51:05.144577 26630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eba364bdc478484c9b1cac5e425313b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cc09cee1327343eead6b474c5aafbbd7"
I20260619 04:51:05.145094 26630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:51:05.151597 26630 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba364bdc478484c9b1cac5e425313b5 in term 1.
I20260619 04:51:05.153108 26457 leader_election.cc:302] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc09cee1327343eead6b474c5aafbbd7, eba364bdc478484c9b1cac5e425313b5; no voters:
I20260619 04:51:05.153618 26677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:51:05.164388 26677 raft_consensus.cc:686] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [term 1 LEADER]: Becoming Leader. State: Replica: eba364bdc478484c9b1cac5e425313b5, State: Running, Role: LEADER
I20260619 04:51:05.165098 26677 consensus_queue.cc:237] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } }
I20260619 04:51:05.151613 26440 master_runner.cc:186] Error getting master registration for 127.3.243.189:40021: OK, instance_id { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" instance_seqno: 1781844664468752 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:51:05.166280 26440 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 04:51:05.170338 26688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:51:05.170471 26688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:51:05.175232 26691 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba364bdc478484c9b1cac5e425313b5. Latest consensus state: current_term: 1 leader_uuid: "eba364bdc478484c9b1cac5e425313b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.175658 26691 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:05.189301 26695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 04:51:05.191681 26695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 04:51:05.192399 26461 tablet.cc:2369] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:51:05.195816 26581 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } attempt: 5
Jun 19 04:51:05 dist-test-slave-2qp8 krb5kdc[26427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844664, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
I20260619 04:51:05.213949 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.196513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33109 (local address 127.0.0.1:50612)
0619 04:51:05.196939 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:05.196959 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:05.197084 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:05.197604 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:05.197609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:05.198111 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:05.198330 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.198344 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.199847 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.199853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.201380 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:05.201397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.202612 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.202618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:05.202855 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.203569 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:05.203593 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:05.206490 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:05.210944 (+ 4454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.210949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.210954 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.211488 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.211938 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:05.211944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:05.211948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:05.212123 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:05.212884 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:05.212889 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:05.213052 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 04:51:05.213208 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 04:51:05.213570 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306}
I20260619 04:51:05.214970 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.196905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50612 (local address 127.3.243.188:33109)
0619 04:51:05.197070 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.197076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.197095 (+ 19us) server_negotiation.cc:407] Connection header received
0619 04:51:05.197204 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.197209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.197319 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.197411 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.198525 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.199680 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.201559 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.202261 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.202393 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.206712 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.206718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.206723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.206763 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.210744 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.211637 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.211642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.211646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.211803 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.212344 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.212349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.212352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.212684 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.212824 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.214338 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.214576 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 04:51:05.220382 26609 tablet.cc:2369] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:51:05.221851 26581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa: Bootstrap starting.
I20260619 04:51:05.222069 26630 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [term 1 FOLLOWER]: Refusing update from remote peer eba364bdc478484c9b1cac5e425313b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:51:05.223336 26691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:51:05.225917 26581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa: Neither blocks nor log segments found. Creating new log.
I20260619 04:51:05.227165 26581 log.cc:824] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa: Log is configured to *not* fsync() on all Append() calls
I20260619 04:51:05.250649 26581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa: No bootstrap required, opened a new log
I20260619 04:51:05.252626 26663 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba364bdc478484c9b1cac5e425313b5. Latest consensus state: current_term: 1 leader_uuid: "eba364bdc478484c9b1cac5e425313b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.252986 26663 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:05.260859 26699 mvcc.cc:205] Tried to move back new op lower bound from 7298435748726878208 to 7298435748522713088. Current Snapshot: MvccSnapshot[applied={T|T < 7298435748726878208}]
I20260619 04:51:05.268500 26581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } }
I20260619 04:51:05.269011 26581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022b1c71c06749249aa3e8f529db63aa, State: Initialized, Role: FOLLOWER
I20260619 04:51:05.269639 26581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } }
I20260619 04:51:05.270176 26701 mvcc.cc:205] Tried to move back new op lower bound from 7298435748726878208 to 7298435748522713088. Current Snapshot: MvccSnapshot[applied={T|T < 7298435748726878208}]
I20260619 04:51:05.276695 26706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.279071 26581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:51:05.279174 26556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eba364bdc478484c9b1cac5e425313b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "022b1c71c06749249aa3e8f529db63aa"
I20260619 04:51:05.280009 26555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:51:05.279204 26706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:05.281927 26663 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eba364bdc478484c9b1cac5e425313b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.282258 26663 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:05.283849 26555 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [term 1 FOLLOWER]: Refusing update from remote peer eba364bdc478484c9b1cac5e425313b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:51:05.284915 26691 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:51:05.287330 26556 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eba364bdc478484c9b1cac5e425313b5 in term 1.
I20260619 04:51:05.288798 26691 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eba364bdc478484c9b1cac5e425313b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.289052 26691 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:05.290674 26695 catalog_manager.cc:1347] Generated new cluster ID: 995d961aa1d24b5f9524875f7d9cab67
I20260619 04:51:05.290772 26695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:51:05.323727 26660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.190741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.3.243.190:44063)
0619 04:51:05.190932 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.190938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.245424 (+ 54486us) server_negotiation.cc:407] Connection header received
0619 04:51:05.246267 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.246276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.246370 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.246451 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.265451 (+ 19000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.266604 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.270918 (+ 4314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.271952 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.272093 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.303570 (+ 31477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.303579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.303583 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.303624 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.307836 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.309385 (+ 1549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.309398 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.309404 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.309591 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.315185 (+ 5594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.315195 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.315201 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.315702 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.316019 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.322845 (+ 6826us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.323182 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 04:51:05.326905 26706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader eba364bdc478484c9b1cac5e425313b5. Latest consensus state: current_term: 1 leader_uuid: "eba364bdc478484c9b1cac5e425313b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.327217 26706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:05.327677 26711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eba364bdc478484c9b1cac5e425313b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.328122 26711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:05.329890 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.190360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.3.243.189:40021)
0619 04:51:05.190545 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.190551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.252645 (+ 62094us) server_negotiation.cc:407] Connection header received
0619 04:51:05.254061 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.254066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.254143 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.254247 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.275844 (+ 21597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.277165 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.292718 (+ 15553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.293556 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.293679 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.310211 (+ 16532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.310222 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.310230 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.310299 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.314324 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.322250 (+ 7926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.322260 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.322265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.322514 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.326648 (+ 4134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.326663 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.326668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.327067 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.327399 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.329120 (+ 1721us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.329460 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:51:05.332902 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.229720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.3.243.188:33109)
0619 04:51:05.236897 (+ 7177us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.236904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.253145 (+ 16241us) server_negotiation.cc:407] Connection header received
0619 04:51:05.254438 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.254443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.254544 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.254621 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.273655 (+ 19034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.275321 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.290738 (+ 15417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.291493 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.291634 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.316756 (+ 25122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:05.316767 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:05.316774 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:05.316829 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:05.321190 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.327910 (+ 6720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.327921 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.327925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.328092 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:05.330317 (+ 2225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:05.330326 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:05.330338 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:05.330703 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:05.330984 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.332166 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.332467 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7025}
I20260619 04:51:05.377600 26677 sys_catalog.cc:455] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eba364bdc478484c9b1cac5e425313b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eba364bdc478484c9b1cac5e425313b5" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 44063 } } peers { permanent_uuid: "022b1c71c06749249aa3e8f529db63aa" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 40021 } } peers { permanent_uuid: "cc09cee1327343eead6b474c5aafbbd7" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33109 } } }
I20260619 04:51:05.377935 26677 sys_catalog.cc:458] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5 [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:05.379208 26695 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:51:05.381088 26695 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:51:05.407152 26695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eba364bdc478484c9b1cac5e425313b5: Generated new TSK 0
I20260619 04:51:05.408520 26695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:51:05.429605 26721 catalog_manager.cc:1259] Loaded cluster ID: 995d961aa1d24b5f9524875f7d9cab67
I20260619 04:51:05.429791 26721 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa: loading cluster ID for follower catalog manager: success
I20260619 04:51:05.433372 26721 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa: acquiring CA information for follower catalog manager: success
I20260619 04:51:05.435834 26721 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 022b1c71c06749249aa3e8f529db63aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 04:51:05.567046 26660 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.533103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.3.243.190:44063)
0619 04:51:05.533316 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.533323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.564076 (+ 30753us) server_negotiation.cc:407] Connection header received
0619 04:51:05.565316 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.565323 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.565433 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.565509 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.566757 (+ 1248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51266: BlockingRecv error: recv got EOF from 127.0.0.1:51266 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 04:51:05.571837 26658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.547740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.3.243.188:33109)
0619 04:51:05.547931 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.547937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.569829 (+ 21892us) server_negotiation.cc:407] Connection header received
0619 04:51:05.570242 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.570249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.570355 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.570435 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.571537 (+ 1102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50630: BlockingRecv error: recv got EOF from 127.0.0.1:50630 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 04:51:05.574296 26585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.539446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.3.243.189:40021)
0619 04:51:05.539693 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.539700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.569247 (+ 29547us) server_negotiation.cc:407] Connection header received
0619 04:51:05.572384 (+ 3137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.572391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.572484 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.572560 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.574054 (+ 1494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46948: BlockingRecv error: recv got EOF from 127.0.0.1:46948 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:51:05.623759 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.606530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.3.243.189:40021)
0619 04:51:05.606766 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.606772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.614787 (+ 8015us) server_negotiation.cc:407] Connection header received
0619 04:51:05.615400 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.615408 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.615632 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.615804 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:05.616562 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.617869 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.620233 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.621059 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.621165 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.622014 (+ 849us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:05.622800 (+ 786us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:05.622937 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.623222 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.623347 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 04:51:05.668759 26658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.655057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.3.243.188:33109)
0619 04:51:05.655309 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.655316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.666442 (+ 11126us) server_negotiation.cc:407] Connection header received
0619 04:51:05.667018 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.667025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.667161 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.667244 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.668462 (+ 1218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50646: BlockingRecv error: recv got EOF from 127.0.0.1:50646 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:51:05.689538 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.682233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.3.243.189:40021)
0619 04:51:05.682439 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.682449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.683015 (+ 566us) server_negotiation.cc:407] Connection header received
0619 04:51:05.683225 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.683232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.683328 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.683438 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:05.684002 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.685215 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.687349 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.688061 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.688216 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.688704 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:05.688871 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:05.688967 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.689145 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.689226 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 04:51:05.702306 26658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.698175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.3.243.188:33109)
0619 04:51:05.698392 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.698399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.700702 (+ 2303us) server_negotiation.cc:407] Connection header received
0619 04:51:05.700882 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.700890 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.701006 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.701125 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.702052 (+ 927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50658: BlockingRecv error: recv got EOF from 127.0.0.1:50658 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 04:51:05.719226 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.710008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.3.243.189:40021)
0619 04:51:05.710180 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.710186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.712696 (+ 2510us) server_negotiation.cc:407] Connection header received
0619 04:51:05.712903 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.712910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.713010 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.713124 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:05.713635 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.714984 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.716957 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.717646 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.717801 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.718315 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:05.718479 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:05.718589 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.718759 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.718849 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 04:51:05.730006 26658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.727420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.3.243.188:33109)
0619 04:51:05.727618 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.727625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.728457 (+ 832us) server_negotiation.cc:407] Connection header received
0619 04:51:05.728671 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.728678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.728771 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.728875 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.729752 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50674: BlockingRecv error: recv got EOF from 127.0.0.1:50674 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 04:51:05.745185 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.737800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.3.243.189:40021)
0619 04:51:05.738047 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.738053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.738627 (+ 574us) server_negotiation.cc:407] Connection header received
0619 04:51:05.738830 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.738836 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.738952 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.739078 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:05.739558 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.740742 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.742853 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.743583 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.743718 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.744279 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:05.744449 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:05.744554 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.744776 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.744865 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260619 04:51:05.758267 26658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.753734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.3.243.188:33109)
0619 04:51:05.754001 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.754010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.756735 (+ 2725us) server_negotiation.cc:407] Connection header received
0619 04:51:05.756904 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.756911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.757032 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.757161 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.758015 (+ 854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50678: BlockingRecv error: recv got EOF from 127.0.0.1:50678 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
I20260619 04:51:05.777667 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.769480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46990 (local address 127.3.243.189:40021)
0619 04:51:05.769690 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.769696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.770261 (+ 565us) server_negotiation.cc:407] Connection header received
0619 04:51:05.770463 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.770469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.770580 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.770715 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:05.771520 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.772867 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.775069 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.775892 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.776058 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.776620 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:05.776816 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:05.776930 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.777100 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.777192 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260619 04:51:05.787544 26658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.785210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.3.243.188:33109)
0619 04:51:05.785413 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.785420 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.785927 (+ 507us) server_negotiation.cc:407] Connection header received
0619 04:51:05.786166 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.786173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.786269 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.786402 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.787289 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50688: BlockingRecv error: recv got EOF from 127.0.0.1:50688 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:51:05.818454 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.810602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.3.243.189:40021)
0619 04:51:05.810836 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.810845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.811479 (+ 634us) server_negotiation.cc:407] Connection header received
0619 04:51:05.811713 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.811720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.811823 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.811968 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:05.812577 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.813890 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.816127 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.816903 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.817040 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.817615 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:05.817761 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:05.817864 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.818056 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.818151 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 04:51:05.836511 26658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.833918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.3.243.188:33109)
0619 04:51:05.834122 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.834129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.834747 (+ 618us) server_negotiation.cc:407] Connection header received
0619 04:51:05.834996 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.835002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.835101 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.835223 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.836277 (+ 1054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50700: BlockingRecv error: recv got EOF from 127.0.0.1:50700 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:51:05.892846 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.885488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.3.243.189:40021)
0619 04:51:05.885775 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.885782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.886339 (+ 557us) server_negotiation.cc:407] Connection header received
0619 04:51:05.886532 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.886539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.886631 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.886730 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:05.887233 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.888405 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.890570 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.891372 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.891496 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.892012 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:05.892172 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:05.892265 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.892436 (+ 171us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.892523 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 04:51:05.903345 26658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.900909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.3.243.188:33109)
0619 04:51:05.901116 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.901122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.901724 (+ 602us) server_negotiation.cc:407] Connection header received
0619 04:51:05.901936 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.901942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.902038 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.902144 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.903094 (+ 950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50716: BlockingRecv error: recv got EOF from 127.0.0.1:50716 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 04:51:05.980546 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:05.975266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.3.243.189:40021)
0619 04:51:05.975516 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.975523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.975832 (+ 309us) server_negotiation.cc:407] Connection header received
0619 04:51:05.976099 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.976131 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.976211 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.976319 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:05.976735 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.977495 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.978756 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:05.979175 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:05.979308 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:05.979718 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:05.979818 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:05.979924 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:05.980093 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 04:51:05.980195 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260619 04:51:05.988487 26658 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:05.986481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.3.243.188:33109)
0619 04:51:05.986693 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:05.986699 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:05.987022 (+ 323us) server_negotiation.cc:407] Connection header received
0619 04:51:05.987190 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:05.987194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:05.987279 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:05.987360 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:05.988250 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50732: BlockingRecv error: recv got EOF from 127.0.0.1:50732 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 04:51:06.085516 26674 catalog_manager.cc:1259] Loaded cluster ID: 995d961aa1d24b5f9524875f7d9cab67
I20260619 04:51:06.085608 26674 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7: loading cluster ID for follower catalog manager: success
I20260619 04:51:06.088086 26674 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7: acquiring CA information for follower catalog manager: success
I20260619 04:51:06.089823 26674 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cc09cee1327343eead6b474c5aafbbd7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:51:06.125201 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:06.120131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.3.243.189:40021)
0619 04:51:06.120386 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:06.120392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:06.120616 (+ 224us) server_negotiation.cc:407] Connection header received
0619 04:51:06.120846 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:06.120852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:06.120931 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:06.121037 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:06.121487 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.122178 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.123464 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.123862 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.124013 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:06.124494 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:06.124608 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:06.124710 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:06.124864 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 04:51:06.124945 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 04:51:06.138092 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:06.132196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50742 (local address 127.3.243.188:33109)
0619 04:51:06.132440 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:06.132447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:06.132736 (+ 289us) server_negotiation.cc:407] Connection header received
0619 04:51:06.132941 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:06.132947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:06.133108 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:06.133255 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:06.133642 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.134353 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.135631 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.136187 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.136310 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:06.136851 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:06.137493 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:06.137656 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:06.137722 (+ 66us) server_negotiation.cc:299] Negotiation successful
0619 04:51:06.137807 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 04:51:06.159055 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:06.144567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.3.243.189:40021)
0619 04:51:06.145026 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:06.145034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:06.152720 (+ 7686us) server_negotiation.cc:407] Connection header received
0619 04:51:06.153061 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:06.153069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:06.153171 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:06.153263 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:06.153906 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.154730 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.156862 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.157537 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.157642 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:06.158261 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:06.158401 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:06.158500 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:06.158674 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 04:51:06.158754 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260619 04:51:06.159096 26827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:06.144194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51274 (local address 127.3.243.190:44063)
0619 04:51:06.152060 (+ 7866us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:06.152070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:06.152580 (+ 510us) server_negotiation.cc:407] Connection header received
0619 04:51:06.152742 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:06.152751 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:06.153022 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:06.153220 (+ 198us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:06.154012 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.155112 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.156847 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.157237 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.157394 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:06.157957 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:06.158457 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:06.158557 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:06.158663 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 04:51:06.158760 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7691,"thread_start_us":7557,"threads_started":1}
I20260619 04:51:06.159618 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:06.144780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.3.243.188:33109)
0619 04:51:06.146117 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:06.146125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:06.153152 (+ 7027us) server_negotiation.cc:407] Connection header received
0619 04:51:06.153684 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:06.153690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:06.153785 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:06.153877 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:06.154615 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.155518 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.157631 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:06.158172 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:06.158293 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:06.158828 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:06.158952 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:06.159013 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:06.159264 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 04:51:06.159345 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981}
I20260619 04:51:06.164319 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26440
I20260619 04:51:06.165329 26502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:51:06.280753 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 26440
I20260619 04:51:06.301517 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26514
I20260619 04:51:06.302656 26576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:51:06.410656 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 26514
I20260619 04:51:06.430799 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26588
I20260619 04:51:06.431951 26650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:51:06.541582 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 26588
2026-06-19T04:51:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4418 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 04:51:06 dist-test-slave-2qp8 krb5kdc[26835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 04:51:06 dist-test-slave-2qp8 krb5kdc[26835](info): set up 2 sockets
Jun 19 04:51:06 dist-test-slave-2qp8 krb5kdc[26835](info): commencing operation
krb5kdc: starting...
W20260619 04:51:07.696835 4046 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/test-user.keytab.
Jun 19 04:51:07 dist-test-slave-2qp8 krb5kdc[26835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T04:51:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T04:51:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
Entry for principal kudu/127.3.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab.
I20260619 04:51:07.959550 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45435
--webserver_interface=127.3.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:45435,127.3.243.189:35233,127.3.243.188:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 04:51:08.291227 26848 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:51:08.291613 26848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:51:08.291776 26848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:51:08.291929 26848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:51:08.299544 26848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:51:08.299639 26848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:51:08.299769 26848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:51:08.299872 26848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:51:08.299994 26848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:51:08.309983 26848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:45435,127.3.243.189:35233,127.3.243.188:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.190.keytab
--principal=kudu/127.3.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.190:45435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:51:08.311394 26848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:51:08.312824 26848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:51:08.324996 26856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:08.325042 26854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:08.325067 26853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:51:08.326323 26848 server_base.cc:1034] running on GCE node
I20260619 04:51:08.327201 26848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:51:08.328752 26848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:51:08.330015 26848 hybrid_clock.cc:648] HybridClock initialized: now 1781844668329959 us; error 45 us; skew 500 ppm
I20260619 04:51:08.330404 26848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:51:08 dist-test-slave-2qp8 krb5kdc[26835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:51:08.337513 26848 init.cc:377] Logged in from keytab as kudu/127.3.243.190@KRBTEST.COM (short username kudu)
I20260619 04:51:08.339063 26848 webserver.cc:460] Webserver started at http://127.3.243.190:44005/ using document root <none> and password file <none>
I20260619 04:51:08.339807 26848 fs_manager.cc:359] Metadata directory not provided
I20260619 04:51:08.339953 26848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:51:08.340368 26848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:51:08.343523 26848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/data/instance:
uuid: "c3b95990e122485c8e21003b73dd76fa"
format_stamp: "Formatted at 2026-06-19 04:51:08 on dist-test-slave-2qp8"
server_key: "f96a43bed8eb31ee6d89cadeaf6c3ccb"
server_key_iv: "33eeb35e9636ad5f9b9ce3fdb3d41426"
server_key_version: "encryptionkey@0"
I20260619 04:51:08.344482 26848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/wal/instance:
uuid: "c3b95990e122485c8e21003b73dd76fa"
format_stamp: "Formatted at 2026-06-19 04:51:08 on dist-test-slave-2qp8"
server_key: "f96a43bed8eb31ee6d89cadeaf6c3ccb"
server_key_iv: "33eeb35e9636ad5f9b9ce3fdb3d41426"
server_key_version: "encryptionkey@0"
I20260619 04:51:08.349889 26848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 04:51:08.353565 26863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:51:08.354940 26848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 04:51:08.355091 26848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/wal
uuid: "c3b95990e122485c8e21003b73dd76fa"
format_stamp: "Formatted at 2026-06-19 04:51:08 on dist-test-slave-2qp8"
server_key: "f96a43bed8eb31ee6d89cadeaf6c3ccb"
server_key_iv: "33eeb35e9636ad5f9b9ce3fdb3d41426"
server_key_version: "encryptionkey@0"
I20260619 04:51:08.355350 26848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:51:08.374475 26848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:51:08.378455 26848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:51:08.378693 26848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:51:08.397197 26848 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.190:45435
I20260619 04:51:08.397217 26914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.190:45435 every 8 connection(s)
I20260619 04:51:08.398555 26848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/data/info.pb
I20260619 04:51:08.402220 26915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:51:08.404588 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 26848
I20260619 04:51:08.404778 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-0/wal/instance
I20260619 04:51:08.405323 4046 external_mini_cluster.cc:1428] Setting key d3406994f2c11bc447a3e0f4854616e1
I20260619 04:51:08.411664 26915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:08 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.189@KRBTEST.COM" created.
I20260619 04:51:08.438069 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.415714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.3.243.190:45435)
0619 04:51:08.417773 (+ 2059us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:08.417786 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:08.417836 (+ 50us) server_negotiation.cc:407] Connection header received
0619 04:51:08.419168 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:08.419178 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:08.419506 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:08.419716 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:08.421014 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.422889 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.424341 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.425304 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.425503 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:08.429124 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:08.429137 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:08.429157 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:08.429200 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:08.433323 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:08.434074 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:08.434085 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:08.434088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:08.434295 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:08.434674 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:08.434679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:08.434682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:08.435406 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:08.435566 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:08.436153 (+ 587us) server_negotiation.cc:299] Negotiation successful
0619 04:51:08.436515 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"spinlock_wait_cycles":15616,"thread_start_us":1543,"threads_started":1}
I20260619 04:51:08.438102 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.415714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45435 (local address 127.0.0.1:33068)
0619 04:51:08.416718 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.416784 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:08.418788 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:08.419786 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:08.419793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:08.420450 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:08.420781 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:08.420799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.423045 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.423049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:08.424206 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:08.424218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.425532 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.425536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:08.425656 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:08.426373 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:08.426394 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:08.428853 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:08.433461 (+ 4608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:08.433482 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:08.433492 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:08.433919 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:08.434408 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:08.434411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:08.434413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:08.434552 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:08.435535 (+ 983us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:08.435541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:08.435736 (+ 195us) client_negotiation.cc:769] Sending connection context
0619 04:51:08.435985 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 04:51:08.436663 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":152,"thread_start_us":422,"threads_started":1}
I20260619 04:51:08.441354 26915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:51:08.442543 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.442164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37304)
0619 04:51:08.442327 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.442433 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 04:51:08.442814 26866 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
I20260619 04:51:08.444351 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.443973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37320)
0619 04:51:08.444143 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.444229 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 04:51:08.444540 26915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:35233: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab.
Entry for principal kudu/127.3.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab.
I20260619 04:51:08.467068 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.189:35233
--webserver_interface=127.3.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab
--principal=kudu/127.3.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:45435,127.3.243.189:35233,127.3.243.188:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:51:08.493942 26915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } attempt: 1
I20260619 04:51:08.495184 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.494714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37326)
0619 04:51:08.494933 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.495061 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 04:51:08.496289 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.495823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37336)
0619 04:51:08.496031 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.496094 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 04:51:08.496469 26915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:35233: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
I20260619 04:51:08.564726 26915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } attempt: 2
I20260619 04:51:08.566100 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.565576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37346)
0619 04:51:08.565851 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.565968 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 04:51:08.567174 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.566828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37354)
0619 04:51:08.566987 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.567060 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 04:51:08.567346 26915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:35233: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
I20260619 04:51:08.658650 26915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } attempt: 3
I20260619 04:51:08.659986 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.659508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37356)
0619 04:51:08.659729 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.659854 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 04:51:08.660985 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.660648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37372)
0619 04:51:08.660791 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.660870 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 04:51:08.661151 26915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:35233: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
W20260619 04:51:08.797807 26922 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:51:08.798187 26922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:51:08.798310 26922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:51:08.798451 26922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 04:51:08.804397 26915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } attempt: 4
I20260619 04:51:08.805698 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.805118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37386)
0619 04:51:08.805476 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.805566 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
W20260619 04:51:08.806118 26922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:51:08.806232 26922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:51:08.806363 26922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:51:08.806449 26922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:51:08.806576 26922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:51:08.806816 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.806405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37400)
0619 04:51:08.806587 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.806697 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260619 04:51:08.807008 26915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.189:35233: Network error: Client connection negotiation failed: client connection to 127.3.243.189:35233: connect: Connection refused (error 111)
I20260619 04:51:08.816859 26922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:45435,127.3.243.189:35233,127.3.243.188:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.189.keytab
--principal=kudu/127.3.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.189:35233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:51:08.818156 26922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:51:08.819407 26922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:51:08.830879 26927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:08.830814 26928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:08.830814 26930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:51:08.831704 26922 server_base.cc:1034] running on GCE node
I20260619 04:51:08.832607 26922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:51:08.833896 26922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:51:08.835103 26922 hybrid_clock.cc:648] HybridClock initialized: now 1781844668835051 us; error 26 us; skew 500 ppm
I20260619 04:51:08.835417 26922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:51:08 dist-test-slave-2qp8 krb5kdc[26835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:51:08.838433 26922 init.cc:377] Logged in from keytab as kudu/127.3.243.189@KRBTEST.COM (short username kudu)
I20260619 04:51:08.839709 26922 webserver.cc:460] Webserver started at http://127.3.243.189:36935/ using document root <none> and password file <none>
I20260619 04:51:08.840418 26922 fs_manager.cc:359] Metadata directory not provided
I20260619 04:51:08.840502 26922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:51:08.840746 26922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:51:08.843514 26922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/data/instance:
uuid: "759552811bfc4e2d94dfbccc0f5026cf"
format_stamp: "Formatted at 2026-06-19 04:51:08 on dist-test-slave-2qp8"
server_key: "8cae7c41f11d4bdccb962daff550440e"
server_key_iv: "951b0615433a894bb43395e89772a7df"
server_key_version: "encryptionkey@0"
I20260619 04:51:08.844321 26922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/wal/instance:
uuid: "759552811bfc4e2d94dfbccc0f5026cf"
format_stamp: "Formatted at 2026-06-19 04:51:08 on dist-test-slave-2qp8"
server_key: "8cae7c41f11d4bdccb962daff550440e"
server_key_iv: "951b0615433a894bb43395e89772a7df"
server_key_version: "encryptionkey@0"
I20260619 04:51:08.849218 26922 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 04:51:08.852421 26937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:51:08.853801 26922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 04:51:08.853999 26922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/wal
uuid: "759552811bfc4e2d94dfbccc0f5026cf"
format_stamp: "Formatted at 2026-06-19 04:51:08 on dist-test-slave-2qp8"
server_key: "8cae7c41f11d4bdccb962daff550440e"
server_key_iv: "951b0615433a894bb43395e89772a7df"
server_key_version: "encryptionkey@0"
I20260619 04:51:08.854211 26922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:51:08.869397 26922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:51:08.873181 26922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:51:08.873462 26922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:51:08.891925 26922 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.189:35233
I20260619 04:51:08.891943 26988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.189:35233 every 8 connection(s)
I20260619 04:51:08.893260 26922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/data/info.pb
I20260619 04:51:08.896778 26989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:51:08.902904 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 26922
I20260619 04:51:08.903090 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-1/wal/instance
I20260619 04:51:08.903631 4046 external_mini_cluster.cc:1428] Setting key a684566bdb3761f6e1bc0785df7a6e24
I20260619 04:51:08.903156 26989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:08 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:51:08.926537 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.906355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.3.243.190:45435)
0619 04:51:08.906600 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:08.906609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:08.907708 (+ 1099us) server_negotiation.cc:407] Connection header received
0619 04:51:08.910525 (+ 2817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:08.910529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:08.910597 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:08.910683 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:08.912691 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.913324 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.915012 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.915750 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.915945 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:08.919459 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:08.919469 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:08.919477 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:08.919521 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:08.923234 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:08.923969 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:08.923977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:08.923983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:08.924159 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:08.924564 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:08.924572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:08.924577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:08.924968 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:08.925116 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:08.925844 (+ 728us) server_negotiation.cc:299] Negotiation successful
0619 04:51:08.926115 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 04:51:08.927134 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.906750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45435 (local address 127.0.0.1:33070)
0619 04:51:08.907527 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.907573 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:08.910235 (+ 2662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:08.910944 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:08.910958 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:08.911739 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:08.912528 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:08.912553 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.913493 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.913501 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:08.914856 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:08.914868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.916003 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.916008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:08.916251 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:08.916932 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:08.916953 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:08.919227 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:08.923377 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:08.923384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:08.923395 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:08.923811 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:08.924264 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:08.924267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:08.924270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:08.924438 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:08.925087 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:08.925093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:08.925407 (+ 314us) client_negotiation.cc:769] Sending connection context
0619 04:51:08.925642 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 04:51:08.926035 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":308,"threads_started":1}
I20260619 04:51:08.929085 26989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.243.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.243.188@KRBTEST.COM" created.
Jun 19 04:51:08 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:51:08.943164 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.929908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37406)
0619 04:51:08.930129 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.930148 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:08.930257 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:08.931040 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:08.931046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:08.931539 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:08.931762 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:08.931775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.933202 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.933206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:08.934101 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:08.934112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.935000 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.935012 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:08.935189 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:08.935758 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:08.935774 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:08.937669 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:08.940731 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:08.940739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:08.940743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:08.941108 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:08.941495 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:08.941500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:08.941504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:08.941624 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:08.942175 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:08.942182 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:08.942333 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 04:51:08.942483 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 04:51:08.942755 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 04:51:08.943245 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.930017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.3.243.189:35233)
0619 04:51:08.930619 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:08.930624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:08.930648 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:51:08.930688 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:08.930694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:08.930872 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:08.930978 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:08.931950 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.933079 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.934249 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:08.934829 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:08.934937 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:08.937865 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:08.937875 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:08.937887 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:08.937915 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:08.940605 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:08.941235 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:08.941249 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:08.941251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:08.941369 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:08.941722 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:08.941729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:08.941732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:08.942055 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:08.942272 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:08.942623 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 04:51:08.942869 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":459,"thread_start_us":316,"threads_started":1}
I20260619 04:51:08.945847 26989 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:51:08.947006 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.946558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37714)
0619 04:51:08.946739 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.946854 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 04:51:08.947266 26939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
I20260619 04:51:08.948868 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.948532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37718)
0619 04:51:08.948671 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.948758 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 04:51:08.949090 26989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33803: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab.
Entry for principal kudu/127.3.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab.
I20260619 04:51:08.965229 4046 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
/tmp/dist-test-taskAbcqnl/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.188:33803
--webserver_interface=127.3.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab
--principal=kudu/127.3.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.243.148:40149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.243.190:45435,127.3.243.189:35233,127.3.243.188:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 04:51:08.998476 26989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } attempt: 1
I20260619 04:51:08.999707 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:08.999169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37732)
0619 04:51:08.999393 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:08.999524 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 04:51:09.000738 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.000362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37742)
0619 04:51:09.000507 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.000611 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 04:51:09.000938 26989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33803: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
I20260619 04:51:09.069166 26989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } attempt: 2
I20260619 04:51:09.070395 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.069878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37748)
0619 04:51:09.070096 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.070206 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 04:51:09.071498 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.071128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37758)
0619 04:51:09.071276 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.071359 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 04:51:09.071767 26989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33803: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
I20260619 04:51:09.106261 26915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } attempt: 5
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:51:09.120388 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.107015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37420)
0619 04:51:09.107238 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.107262 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:09.107381 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:09.107756 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:09.107760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:09.108168 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:09.108346 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.108354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.109690 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.109694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.110559 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.110572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.111608 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.111613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.111789 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.112496 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:09.112513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:09.114562 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:09.117624 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.117630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.117634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.118023 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.118560 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.118565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.118569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.118723 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.119337 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:09.119342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:09.119508 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 04:51:09.119662 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 04:51:09.119982 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 04:51:09.120379 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.107130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.3.243.189:35233)
0619 04:51:09.107330 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.107336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.107360 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:51:09.107522 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.107527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.107608 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.107712 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.108500 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.109543 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.110724 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.111445 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.111595 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.114756 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.114760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.114764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.114795 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.117491 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.118290 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.118297 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.118302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.118433 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.118832 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.118838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.118841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.119124 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.119378 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.119787 (+ 409us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.119995 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 04:51:09.121555 26915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:51:09.122900 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.122454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37760)
0619 04:51:09.122660 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.122757 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 04:51:09.124140 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.123640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37764)
0619 04:51:09.123826 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.123964 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 04:51:09.124377 26915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33803: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
I20260619 04:51:09.163029 26989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } attempt: 3
I20260619 04:51:09.164314 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.163753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37766)
0619 04:51:09.163983 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.164082 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 04:51:09.165400 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.164984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37780)
0619 04:51:09.165213 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.165295 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20260619 04:51:09.165670 26989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33803: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
I20260619 04:51:09.175645 26915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } attempt: 1
I20260619 04:51:09.176774 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.176304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37794)
0619 04:51:09.176498 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.176597 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 04:51:09.177821 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.177446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37800)
0619 04:51:09.177614 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.177696 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 04:51:09.178078 26915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33803: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
I20260619 04:51:09.246330 26915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } attempt: 2
I20260619 04:51:09.247583 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.247042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37804)
0619 04:51:09.247269 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.247376 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 04:51:09.248672 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.248305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37806)
0619 04:51:09.248463 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.248550 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 04:51:09.248883 26915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33803: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
W20260619 04:51:09.295495 26996 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 04:51:09.295783 26996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 04:51:09.295843 26996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 04:51:09.295920 26996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 04:51:09.303107 26996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 04:51:09.303200 26996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 04:51:09.303254 26996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 04:51:09.303303 26996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 04:51:09.303361 26996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 04:51:09.308899 26989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } attempt: 4
I20260619 04:51:09.309999 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.309527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37818)
0619 04:51:09.309766 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.309853 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 04:51:09.310958 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.310631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37830)
0619 04:51:09.310795 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.310861 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 04:51:09.311152 26989 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33803: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
I20260619 04:51:09.313596 26996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.243.148:40149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.243.190:45435,127.3.243.189:35233,127.3.243.188:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/krb5kdc/kudu_127.3.243.188.keytab
--principal=kudu/127.3.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.243.188:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 03:14:01 UTC on ccycloud.quasar-sbacxj.root.comops.site
build id 4454981
ASAN enabled
I20260619 04:51:09.314872 26996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 04:51:09.316228 26996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 04:51:09.328064 27002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:09.328023 27001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 04:51:09.328141 27004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 04:51:09.329174 26996 server_base.cc:1034] running on GCE node
I20260619 04:51:09.329895 26996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 04:51:09.331240 26996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 04:51:09.332448 26996 hybrid_clock.cc:648] HybridClock initialized: now 1781844669332391 us; error 35 us; skew 500 ppm
I20260619 04:51:09.332770 26996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 04:51:09.335850 26996 init.cc:377] Logged in from keytab as kudu/127.3.243.188@KRBTEST.COM (short username kudu)
I20260619 04:51:09.337337 26996 webserver.cc:460] Webserver started at http://127.3.243.188:33591/ using document root <none> and password file <none>
I20260619 04:51:09.338011 26996 fs_manager.cc:359] Metadata directory not provided
I20260619 04:51:09.338092 26996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 04:51:09.338464 26996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 04:51:09.341410 26996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/data/instance:
uuid: "2a0f4ca522914e94aff32f61b430f1ff"
format_stamp: "Formatted at 2026-06-19 04:51:09 on dist-test-slave-2qp8"
server_key: "96a15b86e15c3a65672ad52e95380703"
server_key_iv: "d403f1b79d5223eaaf93a4ff86069b70"
server_key_version: "encryptionkey@0"
I20260619 04:51:09.342165 26996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/wal/instance:
uuid: "2a0f4ca522914e94aff32f61b430f1ff"
format_stamp: "Formatted at 2026-06-19 04:51:09 on dist-test-slave-2qp8"
server_key: "96a15b86e15c3a65672ad52e95380703"
server_key_iv: "d403f1b79d5223eaaf93a4ff86069b70"
server_key_version: "encryptionkey@0"
I20260619 04:51:09.347159 26996 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 04:51:09.350416 27011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 04:51:09.351693 26996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 04:51:09.351868 26996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/data,/tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/wal
uuid: "2a0f4ca522914e94aff32f61b430f1ff"
format_stamp: "Formatted at 2026-06-19 04:51:09 on dist-test-slave-2qp8"
server_key: "96a15b86e15c3a65672ad52e95380703"
server_key_iv: "d403f1b79d5223eaaf93a4ff86069b70"
server_key_version: "encryptionkey@0"
I20260619 04:51:09.352073 26996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 04:51:09.355104 26915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } attempt: 3
I20260619 04:51:09.356343 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.355731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37846)
0619 04:51:09.355997 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.356124 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 04:51:09.357676 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.357120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37862)
0619 04:51:09.357411 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.357515 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260619 04:51:09.357950 26915 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.243.188:33803: Network error: Client connection negotiation failed: client connection to 127.3.243.188:33803: connect: Connection refused (error 111)
I20260619 04:51:09.365477 26996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 04:51:09.370004 26996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 04:51:09.370287 26996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 04:51:09.388378 26996 rpc_server.cc:307] RPC server started. Bound to: 127.3.243.188:33803
I20260619 04:51:09.388408 27062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.243.188:33803 every 8 connection(s)
I20260619 04:51:09.389680 26996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/data/info.pb
I20260619 04:51:09.393186 27063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 04:51:09.399645 4046 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu as pid 26996
I20260619 04:51:09.399833 4046 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAbcqnl/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781844312210132-4046-0/minicluster-data/master-2/wal/instance
I20260619 04:51:09.400417 4046 external_mini_cluster.cc:1428] Setting key bc8b71accb76104f4d00ff04bf122d29
I20260619 04:51:09.400269 27063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
I20260619 04:51:09.422675 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.403074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33076 (local address 127.3.243.190:45435)
0619 04:51:09.403297 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.403304 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.404220 (+ 916us) server_negotiation.cc:407] Connection header received
0619 04:51:09.405893 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.405899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.405979 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.406074 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.407833 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.408959 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.411009 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.411656 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.411776 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.416011 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.416016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.416019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.416050 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.419306 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.420150 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.420154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.420157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.420365 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.420831 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.420834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.420836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.421199 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.421324 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.422084 (+ 760us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.422319 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 04:51:09.423599 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.402114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.3.243.188:33803)
0619 04:51:09.403128 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.403141 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.403965 (+ 824us) server_negotiation.cc:407] Connection header received
0619 04:51:09.405581 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.405600 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.405875 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.406087 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.407847 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.409422 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.411149 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.411971 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.412207 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.416781 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.416791 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.416807 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.416849 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.420112 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.420957 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.420964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.420968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.421105 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.421526 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.421530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.421532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.421833 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.422113 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.422408 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.422653 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":775,"spinlock_wait_cycles":12672,"thread_start_us":1128,"threads_started":1}
I20260619 04:51:09.423641 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.403129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.190:45435 (local address 127.0.0.1:33076)
0619 04:51:09.404048 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.404100 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:09.405609 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:09.406158 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:09.406166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:09.407093 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:09.407644 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.407668 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.409110 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.409114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.410854 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.410875 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.411860 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.411868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.412007 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.412982 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:09.413013 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:09.415832 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:09.419423 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.419432 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.419444 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.420007 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.420475 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.420480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.420483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.420705 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.421354 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:09.421362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:09.421719 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 04:51:09.421979 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 04:51:09.422469 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":19,"thread_start_us":528,"threads_started":1}
I20260619 04:51:09.425289 27063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.190@KRBTEST.COM
I20260619 04:51:09.438520 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.426045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.189:35233 (local address 127.0.0.1:37436)
0619 04:51:09.426231 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.426249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:09.426357 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:09.426854 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:09.426858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:09.427154 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:09.427291 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.427298 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.428229 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.428234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.429052 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.429062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.429719 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.429724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.429915 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.430542 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:09.430558 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:09.432981 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:09.436048 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.436054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.436058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.436545 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.436940 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.436946 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.436949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.437093 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.437573 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:09.437577 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:09.437724 (+ 147us) client_negotiation.cc:769] Sending connection context
0619 04:51:09.437873 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 04:51:09.438145 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260619 04:51:09.438550 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.426023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.3.243.189:35233)
0619 04:51:09.426239 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.426247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.426319 (+ 72us) server_negotiation.cc:407] Connection header received
0619 04:51:09.426517 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.426522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.426596 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.426693 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.427428 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.428072 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.429188 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.429580 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.429702 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.433160 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.433165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.433168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.433199 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.435922 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.436686 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.436693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.436697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.436823 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.437197 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.437204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.437207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.437459 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.437608 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.437961 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.438172 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 04:51:09.439810 27063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } has no permanent_uuid. Determining permanent_uuid...
I20260619 04:51:09.440068 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.426664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33080 (local address 127.3.243.190:45435)
0619 04:51:09.426848 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.426853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.426868 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:51:09.426912 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.426915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.426961 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.427053 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.428066 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.428773 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.429882 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.430507 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.430638 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.434887 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.434898 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.434901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.434929 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.437426 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.438056 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.438061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.438065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.438201 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.438577 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.438582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.438586 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.438935 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.439221 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.439554 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.439768 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.188@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.243.189@KRBTEST.COM
I20260619 04:51:09.453171 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.440647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37888)
0619 04:51:09.440802 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.440819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:09.440907 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:09.441348 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:09.441352 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:09.441861 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:09.442048 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.442059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.442971 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.442977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.443911 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.443922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.444698 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.444704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.444923 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.445476 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:09.445501 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:09.447513 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:09.450872 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.450875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.450878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.451241 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.451704 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.451707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.451710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.451832 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.452493 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:09.452497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:09.452607 (+ 110us) client_negotiation.cc:769] Sending connection context
0619 04:51:09.452724 (+ 117us) client_negotiation.cc:240] Negotiation successful
0619 04:51:09.452911 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 04:51:09.453426 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.440974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.3.243.188:33803)
0619 04:51:09.441110 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.441113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.441127 (+ 14us) server_negotiation.cc:407] Connection header received
0619 04:51:09.441160 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.441163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.441210 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.441336 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.442182 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.442851 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.444093 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.444550 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.444646 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.447709 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.447717 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.447723 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.447760 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.450751 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.451403 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.451408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.451414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.451590 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.451939 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.451944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.451948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.452338 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.452512 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.452860 (+ 348us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.453083 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 04:51:09.456810 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.441664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.3.243.189:35233)
0619 04:51:09.441825 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.441829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.441844 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:51:09.441886 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.441888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.441935 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.442003 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.443248 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.443899 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.445527 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.445917 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.446005 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.450685 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.450693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.450698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.450734 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.454058 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.454826 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.454829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.454832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.454954 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.455387 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.455390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.455392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.455649 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.455918 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.456237 (+ 319us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.456475 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:51:09.459383 27063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff: Bootstrap starting.
I20260619 04:51:09.461334 4046 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 04:51:09.462844 27063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff: Neither blocks nor log segments found. Creating new log.
I20260619 04:51:09.464195 27063 log.cc:824] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff: Log is configured to *not* fsync() on all Append() calls
I20260619 04:51:09.467206 27063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff: No bootstrap required, opened a new log
I20260619 04:51:09.463761 26890 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c3b95990e122485c8e21003b73dd76fa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33080
I20260619 04:51:09.477753 27063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } }
I20260619 04:51:09.478145 27063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0f4ca522914e94aff32f61b430f1ff, State: Initialized, Role: FOLLOWER
I20260619 04:51:09.478650 27063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } }
I20260619 04:51:09.479504 27069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.479835 27069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:09.480757 27063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [sys.catalog]: configured and running, proceeding with master startup.
W20260619 04:51:09.489692 27080 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 04:51:09.489823 27080 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 04:51:09.487748 26996 master_runner.cc:186] Error getting master registration for 127.3.243.190:45435: OK, instance_id { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" instance_seqno: 1781844668391959 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:51:09.498762 26996 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:51:09.535189 26915 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } attempt: 4
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.190@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
I20260619 04:51:09.551250 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.536015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37902)
0619 04:51:09.536231 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.536252 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:09.536392 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:09.536775 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:09.536782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:09.537292 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:09.537525 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.537537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.539004 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.539010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.540554 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.540569 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.541376 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.541384 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.541594 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.542272 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:09.542290 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:09.544570 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:09.548476 (+ 3906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.548481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.548486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.548998 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.549396 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.549401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.549404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.549561 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.550221 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:09.550226 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:09.550425 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 04:51:09.550568 (+ 143us) client_negotiation.cc:240] Negotiation successful
0619 04:51:09.550852 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 04:51:09.551266 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.536089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.3.243.188:33803)
0619 04:51:09.536356 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.536362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.536387 (+ 25us) server_negotiation.cc:407] Connection header received
0619 04:51:09.536524 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.536530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.536614 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.536736 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.537677 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.538851 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.540696 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.541156 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.541325 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.544751 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.544757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.544762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.544807 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.548350 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.549118 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.549123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.549127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.549266 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.549686 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.549690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.549693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.550096 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.550291 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.550681 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.550913 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260619 04:51:09.557243 26915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa: Bootstrap starting.
I20260619 04:51:09.561277 26915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa: Neither blocks nor log segments found. Creating new log.
I20260619 04:51:09.562536 26915 log.cc:824] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa: Log is configured to *not* fsync() on all Append() calls
I20260619 04:51:09.565874 26915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa: No bootstrap required, opened a new log
I20260619 04:51:09.573354 26915 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } }
I20260619 04:51:09.573761 26915 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b95990e122485c8e21003b73dd76fa, State: Initialized, Role: FOLLOWER
I20260619 04:51:09.574262 26915 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } }
I20260619 04:51:09.575232 27083 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.575588 27083 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:09.576427 26915 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:51:09.576494 26890 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 04:51:09.576727 26890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:51:09.579794 26890 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } }
I20260619 04:51:09.581887 26890 leader_election.cc:288] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [CANDIDATE]: Term 1 election: Requested vote from peers 759552811bfc4e2d94dfbccc0f5026cf (127.3.243.189:35233), 2a0f4ca522914e94aff32f61b430f1ff (127.3.243.188:33803)
I20260619 04:51:09.582973 27038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3b95990e122485c8e21003b73dd76fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2a0f4ca522914e94aff32f61b430f1ff"
I20260619 04:51:09.583487 27038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:51:09.588490 27094 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 04:51:09.589285 27038 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b95990e122485c8e21003b73dd76fa in term 1.
I20260619 04:51:09.589978 26865 leader_election.cc:302] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a0f4ca522914e94aff32f61b430f1ff, c3b95990e122485c8e21003b73dd76fa; no voters:
I20260619 04:51:09.590456 27083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [term 1 FOLLOWER]: Leader election won for term 1
I20260619 04:51:09.592725 26848 master_runner.cc:186] Error getting master registration for 127.3.243.189:35233: OK, instance_id { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" instance_seqno: 1781844668886794 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 04:51:09.603821 26848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 04:51:09.604400 27083 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [term 1 LEADER]: Becoming Leader. State: Replica: c3b95990e122485c8e21003b73dd76fa, State: Running, Role: LEADER
I20260619 04:51:09.605168 27083 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } }
I20260619 04:51:09.609524 27098 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3b95990e122485c8e21003b73dd76fa. Latest consensus state: current_term: 1 leader_uuid: "c3b95990e122485c8e21003b73dd76fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.610010 27098 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:09.610455 26989 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } attempt: 5
I20260619 04:51:09.612955 27101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 19 04:51:09 dist-test-slave-2qp8 krb5kdc[26835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781844668, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.243.189@KRBTEST.COM for kudu/127.3.243.188@KRBTEST.COM
W20260619 04:51:09.622267 27017 tablet.cc:2369] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:51:09.628492 26991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.611162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.243.188:33803 (local address 127.0.0.1:37906)
0619 04:51:09.611627 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 04:51:09.611652 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 04:51:09.611795 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 04:51:09.612657 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 04:51:09.612663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 04:51:09.613111 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0619 04:51:09.613329 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.613350 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.614780 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.614786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.616274 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 04:51:09.616290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.617470 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.617477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 04:51:09.617723 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.618447 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 04:51:09.618470 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 04:51:09.621341 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 04:51:09.625478 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.625483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.625487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.626031 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.626553 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 04:51:09.626558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 04:51:09.626562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 04:51:09.626733 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 04:51:09.627407 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 04:51:09.627412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 04:51:09.627571 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 04:51:09.627728 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 04:51:09.628077 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297}
I20260619 04:51:09.629482 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.612064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.3.243.188:33803)
0619 04:51:09.612293 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.612301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.612324 (+ 23us) server_negotiation.cc:407] Connection header received
0619 04:51:09.612387 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.612392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.612493 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.612621 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.613505 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.614578 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.616443 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.617131 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.617249 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.621561 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.621569 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.621574 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.621613 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.625321 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.626235 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.626242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.626245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.626415 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.626882 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.626888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.626891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.627211 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.627347 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.628883 (+ 1536us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.629104 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:51:09.630722 27101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260619 04:51:09.631384 26869 tablet.cc:2369] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 04:51:09.646721 26989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf: Bootstrap starting.
I20260619 04:51:09.650907 26989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf: Neither blocks nor log segments found. Creating new log.
I20260619 04:51:09.652233 26989 log.cc:824] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf: Log is configured to *not* fsync() on all Append() calls
I20260619 04:51:09.658578 27038 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [term 1 FOLLOWER]: Refusing update from remote peer c3b95990e122485c8e21003b73dd76fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:51:09.659075 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.614810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.3.243.190:45435)
0619 04:51:09.615238 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.615244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.631095 (+ 15851us) server_negotiation.cc:407] Connection header received
0619 04:51:09.633062 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.633074 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.633178 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.633301 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.637669 (+ 4368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.638754 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.643233 (+ 4479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.643922 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.644038 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.649398 (+ 5360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.649407 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.649411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.649450 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.653607 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.655040 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.655048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.655052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.655247 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.656028 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.656046 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.656049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.656580 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.657039 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.658067 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.658499 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275}
I20260619 04:51:09.660140 27098 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:51:09.661821 26989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf: No bootstrap required, opened a new log
I20260619 04:51:09.667968 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.621752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.3.243.188:33803)
0619 04:51:09.629933 (+ 8181us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.629938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.635482 (+ 5544us) server_negotiation.cc:407] Connection header received
0619 04:51:09.635632 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.635638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.635709 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.635804 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.642482 (+ 6678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.643577 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.648218 (+ 4641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.648911 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.649009 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.652177 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.652184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.652189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.652225 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.655591 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.664269 (+ 8678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.664277 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.664282 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.664444 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.665296 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.665304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.665309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.665684 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.665975 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.667280 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.667534 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8075,"thread_start_us":19045,"threads_started":1}
I20260619 04:51:09.674067 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.615027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.3.243.189:35233)
0619 04:51:09.615440 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.615452 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.632432 (+ 16980us) server_negotiation.cc:407] Connection header received
0619 04:51:09.632615 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.632622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.632735 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.632840 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.637619 (+ 4779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.638727 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.652919 (+ 14192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:09.653597 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:09.653713 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:09.664338 (+ 10625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 04:51:09.664356 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 04:51:09.664364 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 04:51:09.664403 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 04:51:09.667908 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.668860 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.668865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.668869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.669046 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 04:51:09.670037 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 04:51:09.670060 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 04:51:09.670064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 04:51:09.670438 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 04:51:09.670690 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:09.672868 (+ 2178us) server_negotiation.cc:299] Negotiation successful
0619 04:51:09.673446 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235}
I20260619 04:51:09.681411 26989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } }
I20260619 04:51:09.681916 26989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759552811bfc4e2d94dfbccc0f5026cf, State: Initialized, Role: FOLLOWER
I20260619 04:51:09.682531 26989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } }
I20260619 04:51:09.683028 27069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3b95990e122485c8e21003b73dd76fa. Latest consensus state: current_term: 1 leader_uuid: "c3b95990e122485c8e21003b73dd76fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.683322 27069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:09.688336 27098 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3b95990e122485c8e21003b73dd76fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.688650 27098 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:09.690244 27109 mvcc.cc:205] Tried to move back new op lower bound from 7298435766876909568 to 7298435766709129216. Current Snapshot: MvccSnapshot[applied={T|T < 7298435766876909568}]
I20260619 04:51:09.693923 27101 catalog_manager.cc:1347] Generated new cluster ID: 11618e6d6a5f4afdaaf63a4c6beda2d0
I20260619 04:51:09.694023 27101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 04:51:09.698282 27111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.698750 27111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:09.700865 26989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 04:51:09.701231 26963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 04:51:09.705168 26963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [term 1 FOLLOWER]: Refusing update from remote peer c3b95990e122485c8e21003b73dd76fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 04:51:09.705921 26964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3b95990e122485c8e21003b73dd76fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "759552811bfc4e2d94dfbccc0f5026cf"
I20260619 04:51:09.710379 26964 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b95990e122485c8e21003b73dd76fa in term 1.
I20260619 04:51:09.711779 27083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 04:51:09.713567 27069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3b95990e122485c8e21003b73dd76fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.713845 27069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:09.741173 27111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3b95990e122485c8e21003b73dd76fa. Latest consensus state: current_term: 1 leader_uuid: "c3b95990e122485c8e21003b73dd76fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.741531 27111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:09.742621 27119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3b95990e122485c8e21003b73dd76fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.743002 27119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf [sys.catalog]: This master's current role is: FOLLOWER
I20260619 04:51:09.771826 27083 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3b95990e122485c8e21003b73dd76fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b95990e122485c8e21003b73dd76fa" member_type: VOTER last_known_addr { host: "127.3.243.190" port: 45435 } } peers { permanent_uuid: "759552811bfc4e2d94dfbccc0f5026cf" member_type: VOTER last_known_addr { host: "127.3.243.189" port: 35233 } } peers { permanent_uuid: "2a0f4ca522914e94aff32f61b430f1ff" member_type: VOTER last_known_addr { host: "127.3.243.188" port: 33803 } } }
I20260619 04:51:09.772305 27083 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa [sys.catalog]: This master's current role is: LEADER
I20260619 04:51:09.778774 27101 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 04:51:09.781000 27101 catalog_manager.cc:1504] Loading token signing keys...
I20260619 04:51:09.800143 27101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3b95990e122485c8e21003b73dd76fa: Generated new TSK 0
I20260619 04:51:09.801640 27101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 04:51:09.839318 27128 catalog_manager.cc:1259] Loaded cluster ID: 11618e6d6a5f4afdaaf63a4c6beda2d0
I20260619 04:51:09.839429 27128 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf: loading cluster ID for follower catalog manager: success
I20260619 04:51:09.843146 27128 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf: acquiring CA information for follower catalog manager: success
I20260619 04:51:09.845595 27128 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 759552811bfc4e2d94dfbccc0f5026cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 04:51:09.956466 26919 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:09.923206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.3.243.190:45435)
0619 04:51:09.923424 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.923431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.952829 (+ 29398us) server_negotiation.cc:407] Connection header received
0619 04:51:09.954792 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.954799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.954910 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.954993 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.956190 (+ 1197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33110: BlockingRecv error: recv got EOF from 127.0.0.1:33110 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 04:51:09.960395 27066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:09.936433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.3.243.188:33803)
0619 04:51:09.936615 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.936623 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.958250 (+ 21627us) server_negotiation.cc:407] Connection header received
0619 04:51:09.958891 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.958897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.958985 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.959052 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.960138 (+ 1086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37926: BlockingRecv error: recv got EOF from 127.0.0.1:37926 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 04:51:09.961236 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:09.929427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.3.243.189:35233)
0619 04:51:09.929608 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.929614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:09.957705 (+ 28091us) server_negotiation.cc:407] Connection header received
0619 04:51:09.957868 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:09.957874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:09.957975 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:09.958096 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:09.961007 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37466: BlockingRecv error: recv got EOF from 127.0.0.1:37466 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 04:51:10.008494 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:09.993187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37476 (local address 127.3.243.189:35233)
0619 04:51:09.993378 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:09.993385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.000676 (+ 7291us) server_negotiation.cc:407] Connection header received
0619 04:51:10.001210 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.001217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.001447 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.001598 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.002291 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.003404 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.005469 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.006138 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.006234 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.006990 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.007676 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.007813 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.008039 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.008156 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 04:51:10.054227 27066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:10.041617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.3.243.188:33803)
0619 04:51:10.041825 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.041831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.051659 (+ 9828us) server_negotiation.cc:407] Connection header received
0619 04:51:10.052666 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.052672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.052813 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.052892 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:10.053970 (+ 1078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37942: BlockingRecv error: recv got EOF from 127.0.0.1:37942 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 04:51:10.106647 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.091563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.3.243.189:35233)
0619 04:51:10.091791 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.091797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.099209 (+ 7412us) server_negotiation.cc:407] Connection header received
0619 04:51:10.099753 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.099761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.099888 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.099954 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.100762 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.102025 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.104245 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.105012 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.105114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.105754 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.105933 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.106011 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.106231 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.106333 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260619 04:51:10.124395 27066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:10.121283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.3.243.188:33803)
0619 04:51:10.121532 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.121540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.122058 (+ 518us) server_negotiation.cc:407] Connection header received
0619 04:51:10.122360 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.122367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.122470 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.122548 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:10.124095 (+ 1547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37950: BlockingRecv error: recv got EOF from 127.0.0.1:37950 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:51:10.150453 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.142214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.3.243.189:35233)
0619 04:51:10.142543 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.142552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.143483 (+ 931us) server_negotiation.cc:407] Connection header received
0619 04:51:10.143706 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.143712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.143803 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.143903 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.144466 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.145725 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.147487 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.148322 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.148497 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.149288 (+ 791us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.149484 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.149571 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.149862 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.149970 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
W20260619 04:51:10.163684 27066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:10.161814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.3.243.188:33803)
0619 04:51:10.162010 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.162019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.162043 (+ 24us) server_negotiation.cc:407] Connection header received
0619 04:51:10.162304 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.162310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.162446 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.162560 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:10.163441 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37954: BlockingRecv error: recv got EOF from 127.0.0.1:37954 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 04:51:10.182019 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.173233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.3.243.189:35233)
0619 04:51:10.173484 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.173494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.173515 (+ 21us) server_negotiation.cc:407] Connection header received
0619 04:51:10.173780 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.173788 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.173898 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.174008 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.174837 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.176427 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.179000 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.179908 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.180037 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.180910 (+ 873us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.181077 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.181156 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.181464 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.181611 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260619 04:51:10.192523 27066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:10.190095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.3.243.188:33803)
0619 04:51:10.190395 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.190401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.190423 (+ 22us) server_negotiation.cc:407] Connection header received
0619 04:51:10.190714 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.190723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.190844 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.190965 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:10.192213 (+ 1248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37970: BlockingRecv error: recv got EOF from 127.0.0.1:37970 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
I20260619 04:51:10.214043 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.205398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.3.243.189:35233)
0619 04:51:10.205680 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.205690 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.208786 (+ 3096us) server_negotiation.cc:407] Connection header received
0619 04:51:10.209011 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.209017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.209110 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.209219 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.209684 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.210497 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.211987 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.212556 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.212703 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.213175 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.213310 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.213427 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.213581 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.213663 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260619 04:51:10.224071 27066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:10.221741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37974 (local address 127.3.243.188:33803)
0619 04:51:10.221982 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.221991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.222579 (+ 588us) server_negotiation.cc:407] Connection header received
0619 04:51:10.222807 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.222814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.222902 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.223015 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:10.223807 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37974: BlockingRecv error: recv got EOF from 127.0.0.1:37974 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 04:51:10.250078 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.244084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.3.243.189:35233)
0619 04:51:10.244346 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.244358 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.244936 (+ 578us) server_negotiation.cc:407] Connection header received
0619 04:51:10.245210 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.245215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.245295 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.245417 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.245905 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.246734 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.248089 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.248608 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.248773 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.249242 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.249358 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.249472 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.249629 (+ 157us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.249743 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 04:51:10.259505 27066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:10.256904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37984 (local address 127.3.243.188:33803)
0619 04:51:10.257173 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.257183 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.257769 (+ 586us) server_negotiation.cc:407] Connection header received
0619 04:51:10.257987 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.257994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.258096 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.258217 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:10.259229 (+ 1012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37984: BlockingRecv error: recv got EOF from 127.0.0.1:37984 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260619 04:51:10.301290 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.294954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.3.243.189:35233)
0619 04:51:10.295184 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.295193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.295782 (+ 589us) server_negotiation.cc:407] Connection header received
0619 04:51:10.296037 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.296046 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.296218 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.296283 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.296790 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.297760 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.299215 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.299800 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.299934 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.300416 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.300586 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.300692 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.300827 (+ 135us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.300905 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 04:51:10.312532 27066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:10.310286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.3.243.188:33803)
0619 04:51:10.310537 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.310547 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.311082 (+ 535us) server_negotiation.cc:407] Connection header received
0619 04:51:10.311288 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.311294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.311390 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.311500 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:10.312305 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37986: BlockingRecv error: recv got EOF from 127.0.0.1:37986 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 04:51:10.386698 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.380798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37538 (local address 127.3.243.189:35233)
0619 04:51:10.381015 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.381022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.381415 (+ 393us) server_negotiation.cc:407] Connection header received
0619 04:51:10.381663 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.381670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.381791 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.381894 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.382239 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.383365 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.384968 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.385663 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.385807 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.386139 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.386253 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.386324 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.386426 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.386486 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260619 04:51:10.395867 27066 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 04:51:10.393896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.3.243.188:33803)
0619 04:51:10.394107 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.394115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.394417 (+ 302us) server_negotiation.cc:407] Connection header received
0619 04:51:10.394629 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.394633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.394700 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.394778 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 04:51:10.395625 (+ 847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37992: BlockingRecv error: recv got EOF from 127.0.0.1:37992 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 04:51:10.491734 27080 catalog_manager.cc:1259] Loaded cluster ID: 11618e6d6a5f4afdaaf63a4c6beda2d0
I20260619 04:51:10.491848 27080 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff: loading cluster ID for follower catalog manager: success
I20260619 04:51:10.494334 27080 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff: acquiring CA information for follower catalog manager: success
I20260619 04:51:10.496186 27080 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2a0f4ca522914e94aff32f61b430f1ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 04:51:10.532472 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.527261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37552 (local address 127.3.243.189:35233)
0619 04:51:10.527533 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.527539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.527830 (+ 291us) server_negotiation.cc:407] Connection header received
0619 04:51:10.528081 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.528086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.528227 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.528331 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.528760 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.529434 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.530759 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.531177 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.531326 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.531726 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.531856 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.531958 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.532107 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.532207 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260619 04:51:10.543622 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.538237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.3.243.188:33803)
0619 04:51:10.538408 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.538413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.538837 (+ 424us) server_negotiation.cc:407] Connection header received
0619 04:51:10.538990 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.538994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.539162 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.539259 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.539631 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.540374 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.541679 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.542083 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.542169 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.542594 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.543159 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.543265 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.543350 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.543414 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 04:51:10.562959 27066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.549054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.3.243.188:33803)
0619 04:51:10.549666 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.549674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.556182 (+ 6508us) server_negotiation.cc:407] Connection header received
0619 04:51:10.556542 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.556546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.556608 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.556656 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.557275 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.558428 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.560454 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.561115 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.561206 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.562007 (+ 801us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.562120 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.562205 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.562437 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.562514 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465}
I20260619 04:51:10.563282 27234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.548984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33112 (local address 127.3.243.190:45435)
0619 04:51:10.556645 (+ 7661us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.556649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.556664 (+ 15us) server_negotiation.cc:407] Connection header received
0619 04:51:10.556755 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.556759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.556903 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.557042 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.557835 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.558878 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.560845 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.561390 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.561516 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.562066 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.562501 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.562615 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.562824 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.562916 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7564,"thread_start_us":7470,"threads_started":1}
I20260619 04:51:10.564487 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 04:51:10.549072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37566 (local address 127.3.243.189:35233)
0619 04:51:10.549271 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 04:51:10.549277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 04:51:10.557478 (+ 8201us) server_negotiation.cc:407] Connection header received
0619 04:51:10.557681 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 04:51:10.557688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 04:51:10.557788 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 04:51:10.557898 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 04:51:10.558497 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.559685 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.562004 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 04:51:10.562625 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 04:51:10.562748 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 04:51:10.563631 (+ 883us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 04:51:10.563744 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 04:51:10.563833 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 04:51:10.564084 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 04:51:10.564193 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 04:51:10.568384 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26848
I20260619 04:51:10.569339 26910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:51:10.680533 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 26848
I20260619 04:51:10.701793 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26922
I20260619 04:51:10.702930 26984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:51:10.810528 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 26922
I20260619 04:51:10.830665 4046 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu pid 26996
I20260619 04:51:10.831696 27058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 04:51:10.937290 4046 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAbcqnl/build/asan/bin/kudu with pid 26996
2026-06-19T04:51:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4394 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8812 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (358691 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST